Natural language processing is a subfield of artificial intelligence that combines computational linguistics, statistics, machine learning, and deep learning models to allow computers to process human language and understand its context, intent, and sentiment.<\/p>\n
A generic natural language processing (NLP) model is a combination of multiple mathematical and statistical steps. It usually starts with raw text and ends with a model that can predict outcomes. In between, there are multiple steps that include text cleaning, modeling, and hyperparameter tuning.<\/p>\n
Text cleaning, or normalization, is one of the most important steps of any NLP task. It includes removing unwanted data, converting words to their base forms (stemming<\/a> or lemmatization<\/a>), and vectorization.<\/p>\n Vectorization is the process of converting textual data into numerical vectors and is a process that is usually applied once the text is cleaned. It can help improve the execution speed and reduce the training time of your code. In this article, we will discuss some of the best techniques to perform vectorization.<\/p>\n There are three major methods for performing vectorization on text data: CountVectorizer is one of the simplest techniques that is used for converting text into vectors. It starts by tokenizing the document into a list of tokens (words). It selects the unique tokens from the token list and creates a vocabulary of words. Finally, a sparse matrix is created containing the frequency of words, where each row represents different sentences and each column represents unique words.<\/p>\n Python\u2019s scikit-learn has a class named The first step would be to clean the data by converting text into lowercase and removing stopwords.<\/p>\n First, we will import the We can easily interpret the Document Term Matrix, which is the output of CountVectorizer. It contains all the unique words and their frequency count in different sentences. For example, the \u2018bad\u2019 word count is 2 in the second row because it appeared twice in the second review.<\/p>\n There are also many parameters of CountVectorizer class that can help process the data before applying the vectorization process.<\/p>\n Isolating difficult data samples? Comet can do that. Learn more with our PetCam scenario and discover Comet Artifacts.<\/a><\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n TF-IDF or Term Frequency\u2013Inverse Document Frequency<\/strong><\/a>, is a statistical measure that tells how relevant a word is to a document. It combines two metrics \u2014 term frequency and inverse document frequency \u2014 to produce a relevance score.<\/p>\n The Term Frequency<\/strong> is the frequency of a word in a document. It is calculated by dividing the occurrence of a word inside a document by the total number of words in that document.<\/p>\n The Inverse Document Frequency<\/strong> is a measure of how much information a word provides. Words like \u201cthe,\u201d for example, occur very frequently but provide little context or value to a sentence. It is calculated by taking the inverse log of document frequency, that is the proportion of documents that contain a particular word.<\/p>\n TF-IDF scores range from 0 to 1. A score closer to 1 is higher the importance of a word to a document.<\/p>\n Let\u2019s use the same cleaned corpus from the previous example and perform vectorization using TF-IDF.<\/p>\n So, according to the TF-IDF algorithm words, \u2018bad\u2019 and \u2018good\u2019 are the most important words in the whole document, which is some intent to be true. Since, our data contains reviews impact of words that represent a sentiment (good, bad, etc.) tends to be higher.<\/p>\n Unlike a bag of words, TF-IDF not only works upon frequency but also retrieves their importance towards the document.<\/p>\n Word2Vec is a word embedding technique that makes use of neural networks to convert words into corresponding vectors in a way that semantically similar vectors are close to each other in N-dimensional space, where N refers to the dimensions of the vector. This technique was first implemented by Tomas Mikolov<\/a> at Google<\/a> back in 2013.<\/p>\n Word2Vec has the ability to maintain semantic relations between words. It can be understood by a simple example where if we have a \u201cking\u201d vector and we remove the vector \u201cman\u201d from the king and add a \u201cwomen\u201d vector, then we get a vector close to the \u201cqueen\u201d vector in N-dimensional space.<\/p>\n There are two ways to implement Word2Vec techniques: Skip-Gram and CBOW, both of which we\u2019ll cover below.<\/p>\n Skip-Gram tries to predict several context words from a single input word.<\/p>\n Here w[i] is the input word at position \u2018i\u2019 in the sentence. The output of the model contains two preceding words and two succeeding words with respect to location \u2018i\u2019.<\/p>\n CBOW stands for Continuous Bag of Words, trained to predict a single word from a series of context words. It is the mirror image of the Skip-Gram technique.<\/p>\n Both techniques are good and can generate vectors from the text by considering semantic similarity. Skip-Gram works well with small-size datasets and can find rare words as well. However, CBOW trains faster and can better represent frequent words. According to the original paper, CBOW takes a few hours to train whereas Skip-Gram needs a few days to understand patterns from the data.<\/p>\n The default dimension of word embeddings is 100. we can increase or decrease this number by changing Word2Vec offers different built-in functions to get more insights into the data. Vectorization is an important step that is mostly gets ignored by most beginner data scientists. It is a crucial phase of text cleaning that converts your text data into vectors. Later, these vectors are used for training a machine learning algorithm for generating useful predictions. In short, we can say that vectorization is as important as removing unwanted data from the raw text or training an ml model on the data.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":" Photo by Surendran MP on Unsplash Natural language processing is a subfield of artificial intelligence that combines computational linguistics, statistics, machine learning, and deep learning models to allow computers to process human language and understand its context, intent, and sentiment. A generic natural language processing (NLP) model is a combination of multiple mathematical and statistical […]<\/p>\n","protected":false},"author":47,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[120],"class_list":["post-7852","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\nVectorization techniques<\/h1>\n
\n1. CountVectorizer
\n2. TF-IDF
\n3. Word2Vec<\/p>\n1. CountVectorizer<\/h2>\n
CountVectorizer<\/code> that provides a simple implementation for performing vectorization on text data. Let\u2019s create a sample corpus of food reviews and convert them into vectors using CountVectorizer<\/code>.<\/p>\ncorpus = ['Food is Bad',\n 'Bad Service Bad Food',\n 'Food is Good',\n 'Good Service With Good Food.',\n 'Service is Bad but Food is Good.' ]<\/span><\/pre>\nimport nltk\nnltk.download('stopwords')\n\nfrom nltk.corpus import stopwords\nenglish_stopwords = set(stopwords.words('english'))\n\ncorpus = ['Food is Bad',\n 'Bad Service Bad Food',\n 'Food is Good',\n 'Good Service With Good Food.',\n 'Service is Bad but Food is Good.']\n\ncleaned_corpus = []\nfor sent in corpus:\n sent = sent.lower()\n cleaned_corpus.append(' '.join([word for word in sent.split() if word not in english_stopwords]))\nprint(cleaned_corpus)<\/pre>\ncleaned_corpus = ['food bad',\n 'bad service bad food',\n 'food good',\n 'good service good food.',\n 'service bad food good.'\n ]<\/span><\/pre>\nCountVectorizer<\/code> class:<\/p>\nfrom sklearn.feature_extraction.text import CountVectorizer\nimport pandas as pd\nvectorizer = CountVectorizer()\nX = vectorizer.fit_transform(cleaned_corpus) ## passing cleaned corpus\ndoc_term_matrix = pd.DataFrame(X.toarray(),columns= vectorizer.get_feature_names())\nprint(doc_term_matrix)<\/pre>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:231\/1*QfThtDCGpofk21Z42EuEQA.png\")
<\/figure>Important parameters of CountVectorizer<\/h2>\n
\n
lowercase<\/strong><\/code> : convert all characters to lowercase.<\/li>\nstop_words<\/strong><\/code> : remove stopwords from the text before vectorization. {english},list, default=None. <\/em><\/strong>If \u2018english\u2019, a built-in stop word list is used.<\/li>\nstrip_accents<\/strong><\/code> : {\u2018ascii\u2019, \u2018unicode\u2019}, default=None, <\/em><\/strong>Remove accents and perform other character normalization during the preprocessing step.<\/li>\nngram_range<\/strong><\/code> : tuple (min_n, max_n), default=(1, 1), <\/em><\/strong>range of n-values for different word n-grams to be extracted.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n2. TF-IDF<\/h2>\n
from sklearn.feature_extraction.text import TfidfVectorizer<\/span><\/pre>\ncorpus = ['food bad',\n 'bad service bad food',\n 'food good',\n 'good service good food.',\n 'service bad food good.'\n ]<\/span><\/pre>\nfrom sklearn.feature_extraction.text import TfidfVectorizer\ntf_idf = TfidfVectorizer()\nvectors = tf_idf.fit_transform(corpus)<\/pre>\n<\/div>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:401\/1*7Ekk49LH0U6qsmF5aNuGDw.png\")
<\/figure>3. Word2Vec<\/h2>\n
king - man + women = queen<\/span><\/pre>\n1. Skip-Gram<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:462\/1*EqJ3OlSF97cPO0qoGUCX8A.png\")
<\/figure>2. CBOW<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*jvvkSPkcbFh3m0p4PJcJMw.png\")
<\/figure>import re\nimport nltk\nfrom nltk.corpus import stopwords\nfrom nltk.stem import WordNetLemmatizer\nwordnet_lemmatizer = WordNetLemmatizer()\n\n## Loading dataset from the github repository raw url\ndf = pd.read_csv('https:\/\/raw.githubusercontent.com\/Abhayparashar31\/datasets\/master\/twitter.csv')\n\n## cleaning the text with the help of an external python file containing cleaning function\ncorpus = []\nfor i in range(0,len(df)): #we have 1000 reviews\n corpus.append(clean_text(df['text'][i])) ## 'clean_text' is a separate python file containing a function for cleaning this data. You can find it here : https:\/\/gist.github.com\/Abhayparashar31\/81997c2e2268338809c46a220d08649f\ncorpus_splitted = [i.split() for i in corpus]\n\n## Generating Word Embeddings\nfrom gensim import models\nw2v = models.Word2Vec(corpus_splitted)\n\n## vector representation of word 'flood'\nprint(w2v['flood'])\n\n## 5 most similar words for word 'flood'\nprint(w2v.wv.most_similar('flood')[:5)<\/pre>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*0PfwgyIpZAoJ5DQztaK30A.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:333\/1*yQ0Yl7NsdPGmWIP1JTXcrQ.png\")
<\/figure>vector_size<\/code> parameter. You can find more details about different parameters here<\/strong><\/a>.<\/p>\nmost_simiar<\/code> is one of them that extracts words that have the highest semantic similarity to the input word. Picture 3.4 <\/em>is a screenshot of the 5 most similar words to the word \u2018flood<\/em>\u2019 based on our data. If we validate the output words [injury, death, people, world, life<\/em>] are somehow related to \u2018flood\u2019<\/em>. This clearly indicates that our algorithm has done its job.<\/p>\nRecommended readings<\/h2>\n
1. Efficient Estimation of Word Representations in Vector Space<\/strong>\n[word2vec original paper<\/em><\/a>]<\/span>2. Text Similarity in Vector Space Models: A Comparative Study<\/strong>\n[By Omid Shahmirzadi and others<\/a>]<\/span><\/pre>\nConclusion<\/h2>\n