Natural language refers to the medium we humans use to communicate with each other, and processing simply means the conversion of data into a readable form. In short, natural language processing is a way to provide computers with the ability to understand and communicate in human language.<\/p>\n
NLP is a branch of AI that uses text data as input and return models that can understand and generate insights from new text data. One of the most important steps of creating these models is converting raw text data into a much better-cleaned version that contains only useful information. In this blog, we will look at some techniques to perfectly clean text data for natural language processing.<\/p>\n
It is important to apply each step in the same serial manner as mentioned below, otherwise, you could end up losing lots of useful data.<\/p>\n
Cleaning Data<\/h1>\nNormalize Text<\/h2>\n
It is very common for any text data to have words that follow a certain capitalization like camel case, title case, sentence case, etc., or some mis-capitalized words (eg: pYthOn). Both types create problems in analysis thus it is important to normalize the text into Most of the text data you collect from the web may contain some extra spaces between words, before and after a sentence. It is important to remove these before applying any text processing or cleaning technique to the data.<\/p>\n Unwanted data refers to certain parts of the text that don\u2019t add any value in analysis and model building. For example hashtags, HTML tags, mentions, emails, URLs, phone numbers, or some special combination of characters. We can remove these completely from our text data or replace them with their representative word.<\/p>\n HTML Tags Emails URLs Relying on traditional processes and inconsistent model management can block your team from getting models to production. Building an MLOps strategy can help. Learn more with our free ebook<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Accented Characters For example, r\u00e9sum\u00e9 and resume are two different words for our model, whereas both of them produce the same meaning. These usually occur when you try to collect data from a web source, or a multilingual source.<\/p>\n An abbreviation is a shortened form of a word, for example: TTL: Talk to you later. These usually occur in social media datasets. It becomes important to replace abbreviations with their full form otherwise our model will not be able to learn proper patterns from the data. You can find the JSON file with the most common abbreviation short form and their full version here on my Github<\/em><\/a> profile.<\/p>\n Special symbols are characters that are not considered either letters or digits. Different symbols, punctuation, and accent marks are considered special symbols. They don\u2019t add any value while modeling thus it is important to remove all of them from the text.<\/p>\n Stopwords are English words that do not add any value to the sentence. For the purpose of analyzing text and building NLP models, these words might not add much value thus it is a best practice to remove all the stopwords before proceeding further for vectorization. Some of the most common stopwords are: the, is, for, when, to, at, etc.<\/p>\n There are many ways to remove stopwords, one of the simplest methods is by using the NLTK library.<\/p>\n Stemming is the process of converting a word to its root by removing suffix and prefix from it. Stemming will reduce \u2018Learning,\u2019 \u2018Learns,\u2019 and \u2018Learned,\u2019 to their root word \u2018Learn.\u2019 The NLTK library offers many stemmers, but out of them all, Porter Stemmer and its upgraded version are mostly used.<\/p>\n Lemmatization is similar to stemming but the difference between the two is that it takes into consideration the morphological analysis of the words that allows us to differentiate between present, past, and indefinite tense.<\/p>\n As a quick recap of the article, the initial step for text cleaning is normalization which converts text into lowercase. The next step includes the use of regular expressions to remove any unwanted data from the text by replacing it with white space or some text initials. The text cleaning process ends by removing stopwords and converting text to its base using stemming or lemmatization.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":" Photo by Afif Kusuma on Unsplash Natural language refers to the medium we humans use to communicate with each other, and processing simply means the conversion of data into a readable form. In short, natural language processing is a way to provide computers with the ability to understand and communicate in human language. NLP is […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[140],"class_list":["post-7264","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\nlowercase.<\/code><\/p>\ntext = 'Python PROGRAMMING LanGUage.'\ntext.lower()\n------------------\npython programming language.<\/em><\/span><\/pre>\nRemove Unnecessary Whitespaces<\/h2>\n
doc = 'python programming language '<\/span>import regex as re\nre.sub(\"\\s+\",\" \",doc)\n------------------------\npython programming language<\/em><\/span><\/pre>\nRemoving Unwanted Data<\/h2>\n
\n<\/strong>HTML Tags start with an <<\/code> followed by tag name, ends with><\/code> .<\/p>\ndoc = '<p> Food is very good and <b>cheap<\/b>.<\/p>'<\/span>import regex as re\nre.sub('<.*?>','',doc)\n-------------------\nFood is very good and cheap.<\/em><\/span><\/pre>\n
\n<\/strong>Gmail is one of the most famous and commonly used service providers for email services. Usually, an email starts with a personalized name followed by some initials like digits, special symbols, etc., then @<\/code> ends with an email service provider. Like dazzleninja_44@gmail.com<\/code> .<\/p>\ndoc = 'you can contact me on my work email dazzleninja_44@gmail.com for any queries.'<\/span>import regex as re\nre.sub(r'([a-z0-9+._-]+@[a-z0-9+._-]+\\.[a-z0-9+_-]+)',\"\", doc)\n---------------------\nyou can contact me on my work email for any queries.\n<\/em><\/span>\"\"\"<\/strong>\n[a-z0-9+._-]+\n@\n[a-z0-9+._-]+\n\\\n.\n[a-z0-9+_-]+\n\"\"\"<\/strong><\/span><\/pre>\n
\n<\/strong>A generic URL contains a protocol, subdomain, domain name, top level domain, and directory path.<\/p>\ndoc = 'follow my medium profile at https:\/\/medium.com\/@abhayparashar31 and subscribe to my email list at https:\/\/abhayparashar31.medium.com\/subscribe'\nimport regex as re\nre.sub(r'(http|https|ftp|ssh):\/\/([\\w_-]+(?:(?:\\.[\\w_-]+)+))([\\w.,@?^=%&:\/~+#-]*[\\w@?^=%&\/~+#-])?', '' , doc)\n--------------------\n\"\"\"\nfollow my medium profile at and subscribe to my email list at.\n\"\"\"\n\n\"\"\"\n(http|https|ftp|ssh)\n:\/\/\n([\\w_-]+(?:(?:\\.[\\w_-]+)+))\n([\\w.,@?^=%&:\/~+#-]*[\\w@?^=%&\/~+#-])?\n\"\"\"<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n<\/strong>Accent marks are symbols used over letters especially vowels to emphasize the pronunciation of a word. These characters cause problems in analysis by increasing the vocabulary size unnecessarily.<\/p>\ndoc = 'r\u00e9sum\u00e9 length is good. resume font is bad.'<\/span>import unicodedata\nunicodedata.normalize('NFKD', doc).encode('ascii', 'ignore').decode('utf-8', 'ignore')\n-----------------------\nresume length is good. resume font is bad.<\/em><\/span><\/pre>\nAbbreviations<\/h2>\n
x = \"it'd've better if less food oil is added.\"<\/span>import json\nabbreviations = json.load(open('PATH<\/a>'))\nfor key in abbreviations:\n if key in x:\n x = x.replace(key,abbreviations[key])\nprint(x)<\/span><\/pre>\nRemove Special Symbols<\/h2>\n
doc = 'Congrats!, David You have won 1000$.'<\/span>import regex as re\nre.sub(r'[^\\w ]+', \"\", doc)\n-----------------------\nCongrats David You have won 1000<\/em><\/span><\/pre>\nStopwords<\/h2>\n
doc = 'this is one of the best action movie i have ever watched.'<\/span>import nltk\nnltk.download('stopwords')\nfrom nltk.corpus import stopwords<\/span>english_stopwords = set(stopwords.words('english'))<\/span>cleaned_doc = ' '.join([word for word in doc.split() if word not in english_stopwords])\nprint(cleaned_doc)\n------------------------\none best action movie ever watched.<\/em><\/span><\/pre>\nStemming<\/h2>\n
# nltk.download('punkt')<\/span>from nltk.tokenize import word_tokenize\nfrom nltk.stem import PorterStemmer\nps = PorterStemmer()<\/span>doc = 'learning learn learned learns'<\/span>text = \" \".join([ps.stem(word) for word in word_tokenize(doc)])\nprint(text)\n------------------\nlearn learn learn learn<\/em><\/span><\/pre>\nLemmatization<\/h2>\n
# nltk.download('wordnet')\n# nltk.download('omw-1.4')<\/span>doc = 'history always repeat itself.'<\/span>from nltk.stem import WordNetLemmatizer\nlemmatizer = WordNetLemmatizer()<\/span>text = \" \".join([lemmatizer.lemmatize(word) for word in word_tokenize(doc)])\nprint('Lemmatization: ',text)\n----------------\nLemmatization<\/em><\/strong>: history always repeat itself.\n<\/em>Stemming<\/em><\/strong>: histori alway repeat itself.<\/em><\/span><\/pre>\nConclusion<\/h2>\n