![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*-MUoHYxWr5DFi5UmJHUePQ.jpeg\")
<\/figure>At its core, the discipline of Natural Language Processing (NLP)<\/strong> tries to make the human language \u201cpalatable\u201d to computers. Many data we analyze as data scientists consist of a corpus of human-readable text. Before we can feed this data into a computer for analysis, we must preprocess it.<\/p>\n In this article, let\u2019s dive deep into the Natural Language Toolkit (NLTK) <\/strong>data processing concepts for NLP data. Before building our model, we will also see how we can visualize this data with Kangas<\/strong><\/a> as part of exploratory data analysis (EDA).<\/p>\n Ultimately, we will create an Email Spam Classifier with TensorFlow and the Keras API<\/strong> and track it with Comet<\/strong><\/a>. In the meantime, if you need to become more familiar with Comet, check out their incredible platform here<\/a>.<\/p>\n NLTK offers excellent tools for developing Python programs that leverage natural language data. It has several text-processing libraries for tokenization, stemming, part-of-speech tagging, semantic reasoning, and many more tasks.<\/p>\n To start using NLTK, you need to install it. To do this, just run Tokenization, also referred to as lexical analysis<\/strong><\/a>, is common practice in NLP. It involves splitting text data into lexical tokens, which allow us to work with smaller pieces of text (phrases, sentences, words, or paragraphs) that are still relevant, even once separated from the rest of the text. A tokenizer <\/strong>is responsible for this process.<\/p>\n There are two ways we can tokenize our text data with NLTK. We can either:<\/p>\n Here, we break down the text into individual words. Among other things, this can help identify frequently occurring words for a given analysis.<\/p>\n NLTK provides Notice <\/strong>that we used We must convert the text data into lowercase so all words are in the same case. If this is avoided, the model could interpret words like stock, Stock, and STOCK as unique tokens.<\/p>\n When we have text with multiple sentences, we can break them into a list of individual sentences.<\/p>\n With sentence tokenizing, we get a list of two sentences from the text.<\/p>\n In most cases, this operation occurs after we tokenize the data. Stop words are words that only add to the fluidity of text or speech but contain no meaningful information relevant to our task or the text itself.<\/p>\n First, we need to download and import the stop words<\/strong> that are provided by NLTK. Alternatively, we can also define our own stop words, but for this example the default list is more than sufficient.<\/p>\n Let\u2019s use our word tokenized sentence above:<\/p>\n Notice the disappearance of stop words like \u2018do\u2019, \u2018so\u2019, \u2018from\u2019<\/em> etc.<\/p>\n Stemming is where we reduce words to their base form or root form, like cutting down tree branches to their stem. Stemming normalizes text and makes processing easier for Information Retrieval (IR) or text mining tasks.<\/p>\n A stemmer will apply rules<\/a> that remove suffixes and prefixes from a word and reduce it to its root. This, in turn, reduces the time complexity and space. There are various types of stemming algorithms:<\/p>\n We will use the Porter stemmer as the commonly preferred choice. The approach is similar if you choose to use the other stemmers, that is, based on what model you are building.<\/p>\n We will feed the stemmer the Notice how the words were cut. It is easy to notice that the resultant words are a bit confusing and appear meaningless. This poses one of the negative effects of stemming. The readability of the text is compromised, and in some cases, it may not even produce the correct root form of the word.<\/p>\n A stemmer can also produce inconsistent results. Let\u2019s look at an example:<\/p>\n We notice that the words \u2018Discovering\u2019 <\/em>and \u2018discoveries\u2019 <\/em>do not end up in the same base form (discover<\/em>). There are two main errors<\/strong> we could encounter while stemming:<\/p>\n In most cases, the above errors can be caused by the following:<\/p>\n To get around these errors, we could use Lemmatizers<\/strong> instead of stemmers.<\/p>\n Hang on for lemmatization!<\/p>\n Part of speech tagging involves labeling the words in the text data according to their part of speech. These parts of speech include:<\/p>\n To see all available tags and their meanings run:<\/p>\n Let\u2019s use the An example of where PoS tagging would be applicable is when there is a need to know a product\u2019s qualities in a review. In this case, we could tag the tokenized data, extract all the adjectives, and evaluate the review\u2019s sentiment.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\nGetting started with the NLTK library<\/h1>\n
pip install nltk<\/code>in your command line or !pip install nltk<\/code> if using an interactive notebook.<\/p>\nTokenization<\/h1>\n
\n
Tokenize by word<\/strong><\/h2>\n
word_tokenizer<\/code> that allows us to split text into words. Let\u2019s see how:<\/p>\nimport<\/span> nltk\nfrom<\/span> nltk.tokenize import<\/span> word_tokenize\n\n# Text to tokenize<\/span>\ntext_example = \"\"\"We can do so much with Natural Language Processing ranging\nfrom speech recognition, recommendation systems, spam classifiers, and so\nmany more applications.\"\"\"<\/span>\n\n# Tokenizing<\/span>\nword_tokenized_sent = word_tokenize(text_example.casefold())\nprint<\/span>(word_tokenized_sent)\n\n# RESULTS<\/span>\n'''\n\n['we', 'can', 'do', 'so', 'much', 'with', 'natural', 'language',\n'processing', 'ranging', 'from', 'speech', 'recognition', ',',\n'recommendation', 'systems', ',', 'spam', 'classifiers', ',',\n'and', 'so', 'many', 'more', 'applications', '.']\n\n'''<\/span><\/span><\/pre>\ncasefold()<\/code> on the text_example<\/code>. The method ignores whether the letters in the text are uppercase or lowercase and treats all as lowercase. Alternatively, we could use text_example.lower()<\/code> to convert the text into lowercase.<\/p>\nTokenize by sentence<\/strong><\/h2>\n
from<\/span> nltk.tokenize import<\/span> sent_tokenize\n\ntext_example2 = \"\"\"We can do so much with Natural Language Processing ranging from\nspeech recognition, recommendation systems, spam classifiers, and so many more applications.\nThese applications also leverage the power of Machine Learning and Deep Learning.\n\"\"\"<\/span>\n\nsentence_tokenized= sent_tokenize(sentences_example.casefold())\nprint<\/span>(sentence_tokenized)\n\n# RESULTS <\/span>\n'''\n\n['we can do so much with natural language processing ranging from speech recognition, recommendation systems, spam classifiers, and so many more applications.',\n 'these applications also leverage the power of machine learning and deep learning.'\n\n'''<\/span><\/span><\/pre>\nRemoving stop words<\/h1>\n
nltk.download('stopwords'<\/span>) # download the stopwords from nltk<\/span>\nfrom<\/span> nltk.corpus import<\/span> stopwords\n\n# get stopwords in english<\/span>\neng_stopwords= stopwords.words('english'<\/span>)\n\n# filter stopwords with list comprehension<\/span>\ntokens_no_stopwords = [word for<\/span> word in<\/span> word_tokenized_sent if<\/span> word not<\/span> in<\/span> stopwords_eng]\nprint<\/span>(tokens_no_stopwords)\n\n# RESULTS<\/span>\n'''\n\n['much', 'natural', 'language', 'processing', 'ranging', 'speech', 'recognition',\n',', 'recommendation', 'systems', ',', 'spam', 'classifiers', ',', 'many',\n'applications', '.']\n\n'''<\/span><\/span><\/pre>\nStemming<\/h1>\n
\n
from nltk.stem.porter import PorterStemmer<\/code>.<\/li>\nfrom nltk.stem import SnowballStemmer<\/a><\/code>.<\/li>\nfrom nltk.stem.lancaster import LancasterStemmer<\/a><\/code>.<\/li>\n<\/ul>\ntokens_no_stopwords<\/code> list derived from removing the stop words section.<\/p>\nfrom<\/span> nltk.stem.porter import<\/span> PorterStemmer\n\n# Create a stemmer object<\/span>\nstemmer = PorterStemmer()\n\n# Stemming<\/span>\nstemmed_tokens = [stemmer.stem(token) for<\/span> token in<\/span> tokens_no_stopwords]\n\nprint<\/span>(f\"===Unstemmed tokens==== {tokens_no_stopwords}<\/span>\"<\/span>)\nprint<\/span>(f\"===Stemmed tokens==== {stemmed_tokens}<\/span>\"<\/span>)\n\n# RESULTS<\/span>\n'''\n\n===Unstemmed tokens====\n ['much', 'natural', 'language', 'processing', 'ranging', 'speech', 'recognition',\n ',', 'recommendation', 'systems', ',', 'spam', 'classifiers', ',', 'many', 'applications', '.']\n\n===Stemmed tokens====\n ['much', 'natur', 'languag', 'process', 'rang', 'speech', 'recognit', ',', 'recommend',\n 'system', ',', 'spam', 'classifi', ',', 'mani', 'applic', '.']\n\n'''<\/span><\/span><\/pre>\nsome_txt = \"Discovering the wheel is among the best scientific discoveries ever made.\"<\/span>\n\ndef<\/span> stem_text<\/span>(text<\/span>):\n token_lst = word_tokenize(some_txt.casefold())\n token_no_stpw = [word for<\/span> word in<\/span> token_lst if<\/span> word not<\/span> in<\/span> eng_stopwords]\n inco_stemmed_tokens = [stemmer.stem(token) for<\/span> token in<\/span> token_no_stpw]\n\n return<\/span> inco_stemmed_tokens\n\n# RESULT<\/span>\n'''\n\n['discov', 'wheel', 'among', 'best', 'scientif', 'discoveri', 'ever', 'made', '.']\n\n'''<\/span><\/span><\/pre>\n\n
\n
Part of speech (PoS) Tagging<\/h1>\n
\n
nltk.download('averaged_perceptron_tagger'<\/span>)\nnltk.download('tagsets'<\/span>)\nnltk.help<\/span>.upenn_tagset()<\/span><\/pre>\ntokens_no_stopwords<\/code> list derived from removing the stop words section and tag parts of speech.<\/p>\npos_tagged_tokens = nltk.pos_tag(tokens_no_stopwords)\nprint<\/span>(pos_tagged_tokens)\n\n# RESULTS<\/span>\n'''\n\n[('much', 'JJ'), ('natural', 'JJ'), ('language', 'NN'), ('processing', 'NN'),\n ('ranging', 'VBG'), ('speech', 'NN'), ('recognition', 'NN'), (',', ','),\n ('recommendation', 'NN'), ('systems', 'NNS'), (',', ','), ('spam', 'JJ'),\n ('classifiers', 'NNS'), (',', ','), ('many', 'JJ'), ('applications', 'NNS'),\n ('.', '.')]\n\n'''<\/span> <\/span><\/pre>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*BDlyqK7ZbJZ2EdSE52pUxw.png\")
<\/figure>