{"id":7126,"date":"2023-08-14T04:46:47","date_gmt":"2023-08-14T12:46:47","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7126"},"modified":"2025-04-24T17:14:51","modified_gmt":"2025-04-24T17:14:51","slug":"spam-filtering-using-bag-of-words","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/spam-filtering-using-bag-of-words\/","title":{"rendered":"Spam Filtering Using Bag-of-Words"},"content":{"rendered":"\n\n\n\n\n
\n
\"\"<\/picture>
Photo by Daniel Min\u00e1rik<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n\n
\n
\n

In this post, we\u2019re going to employ one simple natural language processing (NLP) algorithm known as bag-of-words<\/em> to classify messages as ham or spam. Using bag of words and feature engineering related to NLP, we\u2019ll get hands-on experience on a small dataset for SMS classification.<\/p>\n

So, what are we waiting for?<\/em><\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n
Source: https:\/\/adage.com\/article\/digitalnext\/holiday-email-practices-avoiding-spam-filter\/306437<\/a><\/figcaption>\n<\/figure>\n

The Problem: Spam Messages<\/h2>\n

Spam emails or messages belong to the broad category of unsolicited messages received by a user. Spam occupies unwanted space and bandwidth, amplifies the threat of viruses like trojans, and in general exploits a user\u2019s connection to social networks.<\/p>\n

Spam can also be used in Denial of Service (DOS) or Distributed Denial of Service (DDOS) attacks. Various techniques are employed to filter out spam messages, usually centered on content-based filtering. This is because specific keywords, links, or websites are repeatedly sent in bulk to users, characterizing them as spam.<\/p>\n

The Solution: Text Classification<\/h2>\n

Comparatively speaking, languages are harder for algorithms to interpret and analyze than numeric data. This is true for a few reasons:<\/p>\n

    \n
  1. Sentences are not of fixed lengths, but most algorithms require a standard input vector size. Thus, padding is required, corresponding to the largest sentence in the corpora.<\/li>\n
  2. ML algorithms cannot understand words as input: hence, each word needs to be represented by some numeric value.<\/li>\n<\/ol>\n

    Bag-of-Words Model<\/h1>\n

    A bag-of-words model allows us to extract features from textual data. As we know, an algorithm doesn\u2019t understand language. Thus, we need to use a numeric representation for the words in the corpus. This numeric representation can later be fed to any algorithm for further analysis.<\/p>\n

    It\u2019s called \u201cbag-of-words\u201d because the order of the words or the structure of the sentence is lost in this model. Only the occurrence or presence of a word matters.<\/p>\n

    We can think of the model in such a way \u2014 we have a big bag, empty at the start, and a vocabulary or a corpus. We pick up words one by one and put them in the bag, adding to the frequency of their occurrence, and then select the most common words as features for passing through our algorithm of choice.<\/p>\n

    Thus, it promotes the view that similar documents consist of similar kinds of words.<\/p>\n

    \n
    \"\"<\/figure>
    <\/picture><\/div>\n
    Source: https:\/\/dudeperf3ct.github.io\/lstm\/gru\/nlp\/2019\/01\/28\/Force-of-LSTM-and-GRU\/<\/a><\/figcaption>\n<\/figure>\n

    Dataset<\/h1>\n

    The dataset that we\u2019re going to use in this article is an SMS spam collection dataset. It contains over 5500 messages in English, with each message in a column, with the corresponding column next to it specifying whether the text is ham or spam.<\/p>\n

    You can find the dataset here<\/a>. The complete source code can be found in this<\/a>repository.<\/p>\n

    Importing the dataset<\/h2>\n

    To import the dataset into a Pandas dataframe, we use the couple of lines written below:<\/p>\n

    import pandas as pd\ndataset = pd.read_csv('spam.csv', encoding='ISO-8859-1');<\/span><\/pre>\n
    Here\u2019s a glimpse of the dataset we are working on. We, later convert the labels into dummy variables.<\/p>\n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/figure>\n
    Pre-requisites<\/h1>\n
      \n
    1. NLTK<\/a> for NLP-related tasks<\/li>\n
    2. NumPy and Pandas<\/a> for mathematical operations<\/li>\n
    3. Scikit-learn for tokenization and tf-idf model<\/li>\n
    4. Matplotlib<\/a> and Seaborn<\/a> for visualisation tasks<\/li>\n<\/ol>\n
      Data Pre-processing<\/h1>\n
      Removing Stopwords<\/h2>\n
      Stopwords refer to the words in the statement which add no specific meaning to it. They often involve prepositions, helping verbs, and articles (i.e. in, the, an, is). Since these add no value to our model, we need to eradicate them.<\/p>\n
      nltk.download('stopwords')\nfrom nltk.corpus import stopwords\nstopwords.words('english')<\/span><\/pre>\n
      Removing non-alphabetic characters<\/h2>\n
      Since we use only words for text classification, we need to get rid of punctuation and numbers. For this, we use string matching or regex in Python. The below regex only preserves alphabetic words, discarding the rest.<\/p>\n
      text = re.sub('[^A-Za-z]', ' ', text)<\/span><\/pre>\n
      Changing all to lower case<\/h2>\n
      The model cannot distinguish between lowercase and uppercase, treating \u2018Text\u2019 and \u2018text\u2019 as different words. We certainly don\u2019t want that; thus, we change the case of all words to lowercase for simplicity.<\/p>\n
      text = text.lower()<\/span><\/pre>\n
      Spelling Corrections<\/h2>\n
      Sometimes, people write abbreviations or misspell words by mistake. To correct these instances, we use the autocorrect package and its spell corrector.<\/p>\n
      from autocorrect import spell\ntext.append(spell(word))<\/span><\/pre>\n
      Stemming and Lemmatization<\/h2>\n
      Words like act, actor, and acting all are for of the same root word (act).same Stemming and lemmatization are techniques used to truncate words in order to to get the stem or the base word.<\/p>\n
      The difference between these is that after stemming, the stem may not be an actual word, whereas lemmatization always produces a real world, which results in better interpretation of the corpora by humans.<\/p>\n
      For example, studies could be stemmed as studi (not a word), but will be lemmatized as study (an existing word).<\/p>\n
      Here\u2019s is a comparison of the dataset after and before stemming.<\/p>\n
      \n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/div>\n<\/figure>\n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/figure>\n
      You can notice the highlighted words are among a few to be stemmed. Also, stemmed words are not actual words most of the time.<\/p>\n
      Visualizing Spam Keywords<\/h2>\n
      Data visualization is a handy way of better understanding the text data involved in our dataset. For example, we can make a wordcloud, which represents most common words in a space, with the size of each word proportional to the frequency of its occurrence. A few other visualization techniques are discussed here<\/a>.<\/p>\n
      from wordcloud import WordCloud\nimport matplotlib.pyplot as plt\nspam_wc = WordCloud(width = 600,height = 512).generate(spam_words)\nplt.figure(figsize = (12, 8), facecolor = 'k')\nplt.imshow(spam_wc)\nplt.show()<\/span><\/pre>\n
      \n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/div>\n
      Wordcloud of spam words from our dataset.<\/figcaption>\n<\/figure>\n
      Feature Engineering<\/h1>\n
      Now we need to perform manipulation on the cleaned, pre-processed dataset to transform it into a form more suitable for applying a machine learning algorithm.<\/p>\n
      Tokenisation<\/h2>\n
      For the bag of words implementation, we use CountVectorizer<\/code> from scikit-learn, which counts the frequency of each word present in our pre-processed dataset, and takes the n most common words as features.<\/p>\n
      \n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/div>\n
      Source: https:\/\/towardsdatascience.com\/natural-language-processing-count-vectorization-with-scikit-learn-e7804269bb5e<\/a><\/figcaption>\n<\/figure>\n
      CountVectorizer<\/code> returns a matrix, where the rows contain the count of messages containing the word, and columns are the top selected features.<\/p>\n
      from sklearn.feature_extraction.text import CountVectorizer\ndata = CountVectorizer(max_features=2000)\nX = data.fit_transform(dataset).toarray()\n<\/span><\/pre>\n
      Since the countvectoriser contains 2000 features, they are hard to depict here. Thus, for an example below,we take the first 25 words of our dataset, tokenise them and select 10 of the most frequently used ones.<\/strong><\/p>\n
      \n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/div>\n<\/figure>\n
      The matrix that represents the frequency of each of these features in our messages(dataset) is given below:<\/p>\n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n
      z is a sparse matrix consisting of mostly zeros.<\/figcaption>\n<\/figure>\n
      Developing the Model<\/h1>\n
      Now that our dataset is ready with its attributes, we pass it through any algorithm of our choice. Here, after splitting the dataset into training and test sets, I\u2019ve used a simple Naive Bayes<\/a> classifier for demonstration. You can use any algorithm of your choice depending on the dataset.<\/p>\n
      from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y)<\/span>from sklearn.naive_bayes import GaussianNB\nclassifier = GaussianNB()\nclassifier.fit(X_train, y_train)\ny_pred = classifier.predict(X_test)<\/span><\/pre>\n
      Results<\/h1>\n
      Let\u2019s see how our simple model works on a test set:<\/p>\n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/figure>\n
      Looks like our model crossed the finish line with a decent accuracy of ~80%. Not something to boast about, but still pretty decent given the simplicity of our model and its drawbacks, which are discussed in the next section. Thus, we can say that our model differentiates between ham and spam with a good confidence level.<\/p>\n
      \n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/div>\n<\/figure>\n
      Here is an image of the confusion matrix depicting the true positives and false positives in the first row and false negatives and true negatives in the next row respectively.<\/p>\n
      Drawbacks of the Bag-of-Words Model<\/h1>\n
      The bag-of-words model assumes that the words are independent. Thus, it doesn\u2019t take into account any relationship between words. Hence, the meaning of sentences is lost.<\/p>\n
      Also, the structure of the sentence has no importance in the eyes of our model Two sentences like \u201cThese clams are good\u201d and \u201cAre these clams good?\u201d mean the same to the of bag-of-words model, though one is a claims and one is a question. Additionally, for a large vocabulary, bag-of-words result in a very high-dimensional vector.<\/p>\n
      Improvements to the above model<\/h1>\n
      A few ways to improve the accuracy of the above model include:<\/p>\n
        \n
      1. Using custom-made stopwords, as per the dataset\u2019s requirements(what language or lingo your dataset use), you can add other words according to the language of the corpus. Some of the text or links are specific to spam mails, or some lingo is added to pass generic spam filters. This can be avoided by analyzing the dataset well and knowing about the structure and content of spam messages specifically, tailoring stopwords to our needs.<\/li>\n
      2. Instead of using uni-grams (individual words), using bi-grams and tri-grams can be beneficial to better understand a message\u2019s meaning. For example, take a message \u2014 \u201cgift card worth millions\u201d. Here, instead of using uni-grams, which would give us \u2018gift\u2019, \u2018card\u2019, \u2018worth\u2019, \u2018millions\u2019, we can use bi-grams that give us \u2018gift card\u2019 or \u2018worth millions\u2019 together as one feature. As you can see, this could clearly indicate a spam message, whereas \u2018gift\u2019, \u2018card\u2019, \u2018worth\u2019, and \u2018millions\u2019 could individually be a part of any day-to-day conversation.<\/li>\n
      3. Apply a vector space model like cosine similarity between messages, and use tf-idf vectorization to better understand the relative weight of a word to that document\u2019s importance. You can read more about this here in this<\/a> article.<\/li>\n<\/ol>\n
        Repository<\/h2>\n
        Here\u2019s the link<\/a> to my repository, where you can find the complete source code for this tutorial. Also, I will keep on adding code on spam filtering using other techniques soon, so stay connected.<\/p>\n
        Sources to get started with NLP<\/h1>\n
        \n
        \n
        <\/div>\n<\/div>\n<\/div>\n
        \n
        \n
        \n