{"id":7413,"date":"2023-09-11T09:23:30","date_gmt":"2023-09-11T17:23:30","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7413"},"modified":"2025-04-24T17:14:17","modified_gmt":"2025-04-24T17:14:17","slug":"tokenization-techniques-in-nlp","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/tokenization-techniques-in-nlp\/","title":{"rendered":"Tokenization Techniques in NLP"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Image by Shivani Rana<\/a> and Srinivas Chakravarthy<\/a><\/figcaption><\/figure>\n

When you learn about a specific topic, there\u2019s always more to it. In the world of data, there\u2019s always more math behind it, more processes to unravel, and more to learn. In this article, I will go through the different techniques you can use for tokenization in NLP.<\/p>\n

Let\u2019s start off with some definitions\u2026<\/p>\n

Definitions<\/h1>\n

Natural Language Processing (NLP)<\/strong> is a computer\/software\/application\u2019s ability to detect and understand human language through speech and text, just as we humans can.<\/p>\n

Tokenization<\/strong> is the process of breaking down or splitting paragraphs and sentences into smaller units so that they can be easily defined to be used for NLP models. The raw text is broken down into smaller units called tokens.<\/p>\n

Delimiter<\/strong> is a sequence of one or more characters that mark the beginning or end of a unit of data.<\/p>\n

Tokenization Techniques<\/h1>\n

If tokenization is done to split up words, it is called word tokenization<\/strong>. If tokenization is done to split up sentences, it is called sentence tokenization<\/strong>.<\/p>\n

Tokenization is the first step when working on your NLP pipeline; it has a domino effect on the rest of your pipeline. Tokens can help us to understand the frequency of a particular word in the data and can be used directly as a vector representing the data. In this way, the string goes from being unstructured into a numerical data structure which helps your NLP pipeline run smoothly.<\/p>\n

There are different types of tokenization techniques, so let\u2019s dive in!<\/p>\n

White Space Tokenization<\/h1>\n

This is known as one of the simplest tokenization techniques as it uses whitespace within the string as the delimiter of words. Wherever the white space is, it will split the data at that point.<\/p>\n

Although this is one of the simplest and fastest tokenization techniques, the key to this is that it only effectively works with languages where white space is used to break apart words and sentences with meaning, such as English.<\/p>\n

This technique can be easily executed using Python\u2019s built-in functions.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

Regular Expression Tokenizer<\/h1>\n

The regular expression tokenizer is a rule-based tokenizer technique and should be used when other techniques are not serving a specific purpose. For example, there may be punctuation in the text that is causing the data to be unclean, therefore it uses regular expression to split a string into substrings.<\/p>\n

This can be easily executed with NLTK<\/a>, a Python toolkit built for working with NLP. The nltk.tokenize.regexp<\/code> module splits the string into substrings. For example:<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n

Different teams have different needs. Comet\u2019s got you covered. Learn how the team at Uber uses Comet\u2019s experiment management to perform real-time model tuning<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

Penn TreeBank Tokenization<\/h1>\n

The Penn TreeBank Tokenization reads in raw text and outputs tokens of classes based on Penn Treebank<\/strong><\/a>, one of the largest treebanks published. The TreeBank gives the semantic and syntactic annotation of a language.<\/p>\n

Similar to the above technique, it uses regular expression to tokenize text which is similar to the tokenization used in the Penn Treebank and is also part of the NLTK<\/a> Python toolkit. For example, you can see the slight difference in the example above and below:<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

SpaCy<\/h1>\n

SpaCy<\/a> is an open-source Python library that can provide great flexibility. It is a modern version of tokenization in NLP, which is faster and can be simply customized. It has the ability to understand large volumes of text that do not need to be segmented using specific rules. For example:<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

Moses Tokenizer<\/h1>\n

The Moses Tokenizer technique, similar to SpaCy, is also a rule-based tokenizer. It has the ability to separate punctuation from words, while still being able to preserve special tokens such as a date, all whilst normalizing characters with the input of segmentation logic. For example:<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

Subword Tokenization<\/h1>\n

The subword tokenization technique is based on the fact that frequently occurring words should be located in the vocabulary, such as \u201cthere\u201d, \u201chelping\u201d, etc. However, words that aren\u2019t that common will be split into frequent sub words. For example, the word \u2018reiterate\u2019 can be split into subwords of \u201cre\u201d, and \u201citerate\u201d. These subwords are used more frequently and the meaning is still kept intact. These subwords are assigned to a unique ID, so that the model can learn better over time during the training phase.<\/p>\n

There are different types of subword tokenization:<\/p>\n