{"id":7417,"date":"2023-09-11T09:53:44","date_gmt":"2023-09-11T17:53:44","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7417"},"modified":"2025-04-24T17:14:15","modified_gmt":"2025-04-24T17:14:15","slug":"bert-state-of-the-art-model-for-natural-language-processing","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/bert-state-of-the-art-model-for-natural-language-processing\/","title":{"rendered":"BERT: State-of-the-Art Model for Natural Language Processing"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n

BERT is among those developments proposed by the Google research team that shifted machine learning standards by demonstrating outstanding results in NLP tasks like question-answering in chatbot applications, computer translation, language interpretation, next sentence prediction, and much, much more.<\/p>\n

Overview<\/h2>\n

BERT (Bidirectional Encoder representation from Transformers) is an open-source machine learning framework designed to assist computers in comprehending the context of ambiguous language and learn to derive knowledge and patterns from the sequence of words.<\/p>\n

What gives BERT an edge over other NLP models is its bidirectional training which is in contrast to the earlier models that looked at a text sequence only in one direction, either from left to right or vice versa. The BERT model is pre-trained using the Wikipedia corpus and the BooksCorpus. It can also be later fine-tuned and adapted to the vocabulary and datasets of our choice.<\/p>\n

Introduction of BERT model<\/h2>\n

BERT is a transformer model that includes an attention mechanism. It learns to derive contextual relations using a series of encoders that derive feature representations or embeddings from the text.<\/p>\n

The use of bidirectional training helps BERT stand out from naive language models that fail to derive the features from the text from both directions simultaneously; hence the factor of correlation between words is missing from those modes.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Self-attention enabled embeddings generated from the BERT model [Source<\/a>]<\/figcaption><\/figure>\n

A max_len<\/code> bounds the input text to the BERT model to train the model. If the value of max_len<\/code> is set to 512, the 512 tokens will be fed to the input layer padded by tokens if the actual tokens are less than 512 or trimmed if the number of tokens exceeds 512.<\/p>\n

Tokenizing sentences and deriving features from the sequence of words longer than the training corpus is very hard to cope with due to the loss of contextual information. In transformers, when the model tries to predict the next word in the sequence, it searches for the positions where the most information is concentrated using the attention blocks. The attention vector acts as a middleman between the input text and the output value. It convolves the input text vector so that some timestamps that have more hidden information are given high importance by masking with 1, else, given no importance, and masked by 0. BERT usually applies self-attention in between the encoder blocks to give importance to important tokens.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
BERT output as contextualized embeddings [Source<\/a>]<\/figcaption>\n<\/figure>\n

The model takes a CLS<\/code> token prepended with the input text, which acts as a classification token. Each of the encoder layers applies self-attention mechanism and passes the derived embeddings through a feedforward network. The base version of the model outputs a vector of size 768 <\/em>which can be used later with a classifier model to predict the corresponding class labels.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

Innovation and academia go hand-in-hand. Listen to our own CEO Gideon Mendels chat with the Stanford MLSys Seminar Series team about the future of MLOps and give the Comet platform a try for free<\/a>!<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

LSTM vs BERT Transformer model<\/h2>\n

LSTM is dead, long live transformers \ud83d\ude00<\/p><\/blockquote>\n

Traditional LSTM models have deeper roots in the NLP world and have been used to solve natural language processing-related tasks. However, LSTMs take more time concerning training the model since they take words sequentially based on timestamps. LSTMs (or rather, RNNs in general), use sequential processing, i.e., processing word-by-word. On the other hand, transformers use non-sequential processing and look at a complete sentence..<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n
Encoder-decoder mechanism in Transformers [Source<\/a>]<\/figcaption>\n<\/figure>\n

On the other hand, transformers have much higher bandwidth and long-term memory. The model predicts the following word in the sequence based on context vectors associated with these source positions and all the previously generated target words with the attention mechanism<\/a>.<\/p>\n

The BERT model, which is deeply based on transformers models, overcomes these two limitations and makes the model faster and deeply bidirectional.<\/p>\n

Different learning strategies of BERT<\/h1>\n

Masked Language Model (MLM)<\/strong><\/h2>\n

BERT takes in sentences with random words replaced with [MASK] tokens and uses this as input text. In training, BERT attempts to predict the original text of the words replaced by [MASK] tokens, based on the pattern of other sequences of non-masked words. This helps the model better understand the domain-specific context of the language-model. The end goal of the BERT model is to predict the same sentence that was input, thereby filling in the masked tokens with the original words.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Masked Language model in BERT [Source<\/strong><\/a>]<\/figcaption>\n<\/figure>\n

The model uses a \u2018softmax\u2019 layer to predict the word from the vocabulary that would best fill in the blank using probability.<\/p>\n

Next Sentence Prediction<\/h2>\n

In this training technique, BERT takes a set of sentences as input and predicts if one sentence will logically follow the other. This helps Bert understand the context across different sentences and applies the attention technique for long-range memory.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Next sentence prediction training strategy followed by BERT [Source<\/a>]<\/figcaption>\n<\/figure>\n

The model distinguishes between the sets of sentences by the use of tokens which are fed to the model in combination with the input set of sentences:<\/p>\n

    \n
  1. Two sentences are separated by a [SEP<\/code>] token and start with a [CLS<\/code>] token.<\/li>\n
  2. A sentence embedding indicating sentence A\/B is mapped to each word of the sentence to understand the correlation between the sentences.<\/li>\n
  3. Each token\u2019s positional embedding is added to indicate its position in the sequence, giving it a bidirectional overview in terms of model training.<\/li>\n<\/ol>\n

    The training of the BERT model involves using both the strategies of Masked Language Model and Next Sentence Prediction simultaneously, with the aim of minimizing the cumulative loss function of the two strategies and getting a better understanding of the overall language.<\/p>\n

    Why are pre-trained models like BERT better for NLP related tasks?<\/h2>\n

    Transfer learning is a technique of re-applying a previously trained ML model on a new task using the past knowledge from a related task in which the model had already been trained before.<\/p>\n

    The main goal of using a pre-trained model is to solve a similar problem using the derived features from the previously-trained model. Instead of building a model from scratch and going through all the initial stages of training the model, it’s often more efficient to re-use the model trained on another problem as a starting point.<\/p>\n

    \n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/div>\n
    Training vs Performance of models following Transfer Learning [Source<\/a>]<\/figcaption>\n<\/figure>\n
      \n
    1. Higher start:<\/strong> The initial knowledge curve of the base model is already higher than the model trained from scratch.<\/li>\n
    2. Higher slope:<\/strong> The source model\u2019s performance and quality of training, as depicted in the graph, are steeper than they otherwise would be.<\/li>\n
    3. Higher asymptote:<\/strong> The consolidated knowledge of the pre-trained model is better than it otherwise would be.<\/li>\n<\/ol>\n

      Fine Tuning BERT model<\/h2>\n
      \n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/div>\n
      Fine-tuning the BERT model<\/figcaption>\n<\/figure>\n

      The base version of BERT is trained on the Wikipedia corpus and BooksCorpus. Note that it might require some domain-specific knowledge of the language to use. To achieve this, it’s important to fine-tune the model and provide context to the model so that it performs well in scenarios where domain-specific knowledge is required.<\/p>\n

      Fine-tuning the model on even a much smaller dataset as compared to the one on which it was originally trained can bring major advantages as well. Depending on your problem at hand, some fine-tuning techniques include:<\/p>\n