close
The Wayback Machine - https://web.archive.org/web/20240905155153/https://www.geeksforgeeks.org/python-stemming-words-with-nltk/
Open In App

Python | Stemming words with NLTK

Last Updated : 15 Apr, 2023
Comments
Improve
Suggest changes
Like Article
Like
Save
Share
Report
News Follow

Stemming is the process of producing morphological variants of a root/base word. Stemming programs are commonly referred to as stemming algorithms or stemmers. A stemming algorithm reduces the words “chocolates”, “chocolatey”, and “choco” to the root word, “chocolate” and “retrieval”, “retrieved”, “retrieves” reduce to the stem “retrieve”.

Prerequisite: Introduction to Stemming

Some more example of stemming for root word "like" include:

-> "likes"
-> "liked"
-> "likely"
-> "liking"

Errors in Stemming: There are mainly two errors in stemming – Overstemming and Understemming. Overstemming occurs when two words are stemmed from the same root that are of different stems. Under-stemming occurs when two words are stemmed from the same root that is not of different stems.

Applications of stemming are:  

  • Stemming is used in information retrieval systems like search engines.
  • It is used to determine domain vocabularies in domain analysis.

Stemming is desirable as it may reduce redundancy as most of the time the word stem and their inflected/derived words mean the same.

Below is the implementation of stemming words using NLTK:

Code #1:  

Python3




# import these modules
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
 
ps = PorterStemmer()
 
# choose some words to be stemmed
words = ["program", "programs", "programmer", "programming", "programmers"]
 
for w in words:
    print(w, " : ", ps.stem(w))


Output: 

program  :  program
programs  :  program
programmer  :  program
programming  :  program
programmers  :  program

Code #2: Stemming words from sentences

Python3




# importing modules
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
 
ps = PorterStemmer()
 
sentence = "Programmers program with programming languages"
words = word_tokenize(sentence)
 
for w in words:
    print(w, " : ", ps.stem(w))


Output : 

Programmers  :  program
program  :  program
with  :  with
programming  :  program
languages  :  language

Code #3: Using reduce():

Algorithm :

  1. Import the necessary modules: PorterStemmer and word_tokenize from nltk, and reduce from functools.
  2. Create an instance of the PorterStemmer class.
  3. Define a sample sentence to be stemmed.
  4. Tokenize the sentence into individual words using word_tokenize.
  5. Use reduce to apply the PorterStemmer to each word in the tokenized sentence, and join the stemmed words back into a string.
  6. Print the stemmed sentence.
install the pip install nltk

Python3




from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
from functools import reduce
 
ps = PorterStemmer()
 
sentence = "Programmers program with programming languages"
words = word_tokenize(sentence)
 
# using reduce to apply stemmer to each word and join them back into a string
stemmed_sentence = reduce(lambda x, y: x + " " + ps.stem(y), words, "")
 
print(stemmed_sentence)
#This code is contrinuted by Pushpa.


Output:

Programm program with program language

Time complexity:
The time complexity of this code is O(nlogn), where n is the length of the input sentence. The tokenizer and stemmer functions have a linear time complexity of O(n), but the reduce function has a logarithmic time complexity of O(logn) since it processes elements in pairs.

Space complexity:
The space complexity of this code is O(n), where n is the length of the input sentence. This is because the reduce function creates a new string object that has the same length as the input sentence. The tokenizer and stemmer functions do not increase the space complexity significantly.



Similar Reads

Introduction to Stemming
Stemming is a method in text processing that eliminates prefixes and suffixes from words, transforming them into their fundamental or root form, The main objective of stemming is to streamline and standardize words, enhancing the effectiveness of the natural language processing tasks. The article explores more on the stemming technique and how to p
9 min read
Python | Lemmatization with NLTK
Lemmatization is a fundamental text pre-processing technique widely applied in natural language processing (NLP) and machine learning. Serving a purpose akin to stemming, lemmatization seeks to distill words to their foundational forms. In this linguistic refinement, the resultant base word is referred to as a "lemma." The article aims to explore t
6 min read
Bag of words (BoW) model in NLP
In this article, we are going to discuss a Natural Language Processing technique of text modeling known as Bag of Words model. Whenever we apply any algorithm in NLP, it works on numbers. We cannot directly feed our text into that algorithm. Hence, Bag of Words model is used to preprocess the text by converting it into a bag of words, which keeps a
4 min read
NLP | How to score words with Execnet and Redis
Distributed word scoring can be performed using Redis and Execnet together. For each word in movie_reviews corpus, FreqDist and ConditionalFreqDist are used to calculate information gain. Using >RedisHashFreqDist and a RedisConditionalHashFreqDist, same thing can be performed with Redis. The scores are then stored in a RedisOrderedDict. In order
4 min read
Top K Nearest Words using Edit Distances in NLP
We can find the Top K nearest matching words to the given query input word by using the concept of Edit/Levenstein distance. If any word is the same word as the query input(word) then their Edit distance would be zero and is a perfect match and so on. So we can find the Edit distances between the query word and the words present in our vocabulary p
3 min read
Finding the Word Analogy from given words using Word2Vec embeddings
In many placement exam rounds, we often encounter a basic question to find word analogies. In the word analogy task, we complete the sentence "a is to b as c is to ___ ", which is often represented as a : b :: c : d and we have to find the word 'd'. A sample question can be like: 'man is to woman as king is to ___'. The human brain can recognize th
3 min read
Finding the Odd Word amongst given words using Word2Vec embeddings
Odd One out the problem is one of the most interesting and goto problems when it comes to testing the logical reasoning skills of an individual. It is often used in many competitive exams and placement rounds as it checks the individual's analytical skills and decision-making ability. In this article, we are going to write a python code that can be
3 min read
NLP | How tokenizing text, sentence, words works
Tokenization in natural language processing (NLP) is a technique that involves dividing a sentence or phrase into smaller units known as tokens. These tokens can encompass words, dates, punctuation marks, or even fragments of words. The article aims to cover the fundamentals of tokenization, it's types and use case. What is Tokenization in NLP?Natu
8 min read
Python | Sort Python Dictionaries by Key or Value
There are two elements in a Python dictionary-keys and values. You can sort the dictionary by keys, values, or both. In this article, we will discuss the methods of sorting dictionaries by key or value using Python. Need for Sorting Dictionary in PythonWe need sorting of data to reduce the complexity of the data and make queries faster and more eff
8 min read
str() vs repr() in Python
In Python, the str() and repr() functions are used to obtain string representations of objects. While they may seem similar at first glance, there are some differences in how they behave. Both of the functions can be helpful in debugging or printing useful information about the object. Python str() In the given example, we are using the str() funct
3 min read
Time Functions in Python | Set 1 (time(), ctime(), sleep()...)
Python has defined a module, "time" which allows us to handle various operations regarding time, its conversions and representations, which find its use in various applications in life. The beginning of time started measuring from 1 January, 12:00 am, 1970 and this very time is termed as "epoch" in Python. Operations on Time in Python Python time.t
4 min read
Type Conversion in Python
Python defines type conversion functions to directly convert one data type to another which is useful in day-to-day and competitive programming. This article is aimed at providing information about certain conversion functions. There are two types of Type Conversion in Python: Python Implicit Type ConversionPython Explicit Type ConversionType Conve
5 min read
Python | Set 3 (Strings, Lists, Tuples, Iterations)
In the previous article, we read about the basics of Python. Now, we continue with some more python concepts. Strings in Python: A string is a sequence of characters that can be a combination of letters, numbers, and special characters. It can be declared in python by using single quotes, double quotes, or even triple quotes. These quotes are not a
3 min read
Convert Decimal to Other Bases in Python
Given a number in decimal number convert it into binary, octal and hexadecimal number. Here is function to convert decimal to binary, decimal to octal and decimal to hexadecimal. Examples: Input : 55 Output : 55 in Binary : 0b110111 55 in Octal : 0o67 55 in Hexadecimal : 0x37 Input : 282 Output : 282 in Binary : 0b100011010 282 in Octal : 0o432 282
2 min read
ord() function in Python
Python ord() function returns the Unicode code from a given character. This function accepts a string of unit length as an argument and returns the Unicode equivalence of the passed argument. In other words, given a string of length 1, the ord() function returns an integer representing the Unicode code point of the character when an argument is a U
3 min read
Plotting graph using Seaborn | Python
This article will introduce you to graphing in Python with Seaborn, which is the most popular statistical visualization library in Python. Installation: The easiest way to install seaborn is to use pip. Type following command in terminal: pip install seaborn OR, you can download it from here and install it manually. Plotting categorical scatter plo
8 min read
Union() function in Python
Python set Union() Method returns a new set that contains all the items from the original set. The union of two given sets is the set that contains all the elements of both sets. The union of two given sets A and B is a set that consists of all the elements of A and all the elements of B such that no element is repeated. The symbol for denoting uni
3 min read
Print all sublists of a list in Python
Given a list, print all the sublists of a list. Python provides efficient and elegant ways to generate and manipulate sublists. In this article, we will explore different methods how to print all sublists of a given list. Examples: Input: list = [1, 2, 3] Output: [[], [1], [1, 2], [1, 2, 3], [2], [2, 3], [3]]Input: [1, 2, 3, 4] Output: [[], [1], [1
3 min read
Analysis of test data using K-Means Clustering in Python
This article demonstrates an illustration of K-means clustering on a sample random data using open-cv library. Pre-requisites: Numpy, OpenCV, matplot-lib Let's first visualize test data with Multiple Features using matplot-lib tool. # importing required tools import numpy as np from matplotlib import pyplot as plt # creating two test data X = np.ra
2 min read
Python set symmetric_difference_update()
The symmetric difference between the two sets is the set of elements that are in either of the sets but not in both of them. Symmetric Difference is marked in Green symmetric_difference() method returns a new set that contains a symmetric difference of two sets. The symmetric_difference_update() method updates the set by calling symmetric_differenc
4 min read
Python dictionary values()
values() is an inbuilt method in Python programming language that returns a view object. The view object contains the values of the dictionary, as a list. If you use the type() method on the return value, you get "dict_values object". It must be cast to obtain the actual list. Python Dictionary values() Method Syntax Syntax: dictionary_name.values(
2 min read
Handling Categorical Data with Bokeh - Python
As a data scientist, you will often come across datasets with categorical data. Categorical data is a type of data that can be divided into distinct categories or groups. For example, a dataset might have a column with the categories "red", "green", and "blue". Handling categorical data can be challenging because it cannot be processed in the same
5 min read
Python | How and where to apply Feature Scaling?
Feature Scaling or Standardization: It is a step of Data Pre Processing that is applied to independent variables or features of data. It helps to normalize the data within a particular range. Sometimes, it also helps in speeding up the calculations in an algorithm. Package Used: sklearn.preprocessing Import: from sklearn.preprocessing import Standa
2 min read
Python - Efficient Text Data Cleaning
Gone are the days when we used to have data mostly in row-column format, or we can say Structured data. In present times, the data being collected is more unstructured than structured. We have data in the form of text, images, audio etc and the ratio of Structured to Unstructured data has decreased over the years. Unstructured data is increasing at
7 min read
Python | Tensorflow nn.sigmoid()
Tensorflow is an open-source machine learning library developed by Google. One of its applications is to develop deep neural networks. The module tensorflow.nn provides support for many basic neural network operations.One of the many activation functions is the sigmoid function which is defined as [Tex]f(x) = 1 / (1 + e^{-x}) [/Tex].Sigmoid functio
3 min read
Python | Tensorflow nn.relu() and nn.leaky_relu()
Tensorflow is an open-source machine learning library developed by Google. One of its applications is to developed deep neural networks. The module tensorflow.nn provides support for many basic neural network operations. An activation function is a function which is applied to the output of a neural network layer, which is then passed as the input
3 min read
Converting Color video to grayscale using OpenCV in Python
OpenCV is a huge open-source library for computer vision, machine learning, and image processing. It can process images and videos to identify objects, faces, or even the handwriting of a human. In this article, we will see how to convert a colored video to a gray-scale format. Approach: Import the cv2 module.Read the video file to be converted usi
1 min read
Python | Image Classification using Keras
Image classification is a method to classify way images into their respective category classes using some methods like : Training a small network from scratchFine-tuning the top layers of the model using VGG16 Let's discuss how to train the model from scratch and classify the data containing cars and planes. Train Data: Train data contains the 200
4 min read
Python next() method
Python's next() function returns the next item of an iterator. Example Let us see a few examples to see how the next() method in Python works. C/C++ Code l_iter = iter(l) print(next(l_iter)) Output1 Note: The .next() method was a method for iterating over a sequence in Python 2. It has been replaced in Python 3 with the next() function, which is ca
4 min read
Python setattr() method
Python setattr() method is used to assign the object attribute its value. The setattr() can also be used to initialize a new object attribute. Also, setattr() can be used to assign None to any object attribute. Example : C/C++ Code class Person: def __init__(self): pass p = Person() setattr(p, 'name', 'kiran') print(f"name: {p.name}") Out
3 min read
Article Tags :
Practice Tags :