With the rise of neural network NLP models, many practitioners are wondering how to best configure a model to perform a particular task. For example, how should I structure a model to perform sentiment analysis? Translation? Essay generation?<\/p>\n
In this post, I will discuss the three dominant model architectures used in NLP today, and when each should be used to maximize your model\u2019s performance and efficiency.<\/p>\n
Architectures<\/h1>\nEncoder-Only<\/strong><\/h2>\nThis type of model takes as input a sequence of words and produces a fixed number of outputs. For example, if I want to use an encoder only architecture to classify the sentiment of a sentence, I will always predict 1 number \u2014 either a 1 (for \u201cpositive\u201d) or a 0 (for \u201cnegative\u201d). The same goes for predicting movie star ratings \u2014 I will still predict 1 number, though it can be either 1, 2, 3, 4, or 5. If I were to do entity detection\u2014 i.e., tagging each word as being a \u201cperson\u201d, \u201cnumber\u201d, \u201clocation\u201d, etc., I could use an encoder-only model to predict the number of tokens equal to the length of the input sequence.<\/p>\n
In general, if you know exactly how many numbers\/words you want your NLP model to output, the encoder-only architecture is likely best. Unlike the next two configurations I will discuss, it neither needs to waste model space trying to figure out how many values need to be outputted nor restricts a model\u2019s ability to look at all words in an input sequence to make a decision. Some examples of popular encoder-only NLP models are BERT (Devlin et al., 2018), RoBERTA (Liu et al., 2019), DeBERTa (He et al., 2020), and ELECTRA (Clark et al., 2020).<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:318\/1*J5IksFwXyIsmrDo2KIjk-w.png\")
<\/figure><\/picture><\/div>\n<\/div>Figure 1: Visualization of an encoder-only architecture. Taken from BERT\u2019s paper: https:\/\/arxiv.org\/pdf\/1910.10683.pdf<\/a><\/figcaption><\/figure>\nDecoder-Only<\/strong><\/h2>\nLike the encoder-only model, the decoder-only model setup also takes as input a sequence of words. However, it differs in that rather than trying to predict a fixed sequence of numbers or tokens, it attempts to actually output a text sequence. This is advantageous if your task\u2019s output is variable \u2014 e.g., if you train a model to answer questions, the answers to \u201cWhat is the capital of Germany?\u201d and \u201cWhat is the General Theory of Relativity?\u201d will have very different lengths.<\/p>\n
However, there is a cost to this output flexibility \u2014 while the encoder-only model gets to make decisions based on seeing every word in the input sequence, the decoder-only model only has partial visibility. In particular, a neuron corresponding to the n<\/em>th word of the input sequence can only make decisions based on the previous n-1<\/em> words \u2014 it cannot look forward. This ends up significantly hurting its performance for a given model size.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\nExisting ML applications may surprise you \u2014 watch our interview with GE Healthcare\u2019s Vignesh Shetty<\/a> to learn how his team is using ML in the healthcare setting.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\nTo provide an example, two of the best models of 2018 were decoder-only GPT (Radford et al., 2018) and encoder-only BERT (Devlin et al., 2018). Despite GPT being 36% larger (150 million parameters to 110 parameters), it actually performed 6% worse on a natural language understanding test called GLUE (75.1 vs 79.6). Some of the most famous decoder-only models are the GPT series made by OpenAI and EleutherAI. This type of model is best used for tasks like freestyle text generation and language modeling.<\/p>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:435\/1*Mis6T15sJnRuC0iwpm5pRQ.png\")
<\/figure><\/picture><\/div>\nFigure 2: Visualization of a decoder-only architecture. Taken from T5\u2019s paper: https:\/\/arxiv.org\/pdf\/1910.10683.pdf<\/a><\/figcaption>\n<\/figure>\nEncoder-Decoder Model<\/strong><\/h2>\nThis type of model combines the encoder-only and decoder-only setups to try and get the best of both worlds. The model starts by taking an input sequence and passing it through an encoder. This encoder outputs what we call a context vector \u2014 a set of numbers of fixed size that basically represents the model\u2019s understanding of the input (think of this as the state of your brain\u2019s neurons as you are listening to someone\u2019s request). The decoder then takes this context vector and uses it to output the result.<\/p>\n
This setup ends up preserving the decoder\u2019s ability to have variable length generations while reducing the inefficiencies associated with only having partial visibility of the sequence. This type of model works quite well for machine translation (e.g., translating from Language A to Language B, where a language could be a human language, a programming language, etc.), summarization, and question answering. Examples of models using this architecture include T5 (Raffel et al., 2019), BART (Liu et al., 2019), PEGASUS (Zhang et al., 2019), and Meena (Adiwardana et al., 2020).<\/p>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:383\/1*hGYro_xh2-JSAqe8KoFjww.png\")
<\/figure>
This type of model takes as input a sequence of words and produces a fixed number of outputs. For example, if I want to use an encoder only architecture to classify the sentiment of a sentence, I will always predict 1 number \u2014 either a 1 (for \u201cpositive\u201d) or a 0 (for \u201cnegative\u201d). The same goes for predicting movie star ratings \u2014 I will still predict 1 number, though it can be either 1, 2, 3, 4, or 5. If I were to do entity detection\u2014 i.e., tagging each word as being a \u201cperson\u201d, \u201cnumber\u201d, \u201clocation\u201d, etc., I could use an encoder-only model to predict the number of tokens equal to the length of the input sequence.<\/p>\n
In general, if you know exactly how many numbers\/words you want your NLP model to output, the encoder-only architecture is likely best. Unlike the next two configurations I will discuss, it neither needs to waste model space trying to figure out how many values need to be outputted nor restricts a model\u2019s ability to look at all words in an input sequence to make a decision. Some examples of popular encoder-only NLP models are BERT (Devlin et al., 2018), RoBERTA (Liu et al., 2019), DeBERTa (He et al., 2020), and ELECTRA (Clark et al., 2020).<\/p>\n Like the encoder-only model, the decoder-only model setup also takes as input a sequence of words. However, it differs in that rather than trying to predict a fixed sequence of numbers or tokens, it attempts to actually output a text sequence. This is advantageous if your task\u2019s output is variable \u2014 e.g., if you train a model to answer questions, the answers to \u201cWhat is the capital of Germany?\u201d and \u201cWhat is the General Theory of Relativity?\u201d will have very different lengths.<\/p>\n However, there is a cost to this output flexibility \u2014 while the encoder-only model gets to make decisions based on seeing every word in the input sequence, the decoder-only model only has partial visibility. In particular, a neuron corresponding to the n<\/em>th word of the input sequence can only make decisions based on the previous n-1<\/em> words \u2014 it cannot look forward. This ends up significantly hurting its performance for a given model size.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Existing ML applications may surprise you \u2014 watch our interview with GE Healthcare\u2019s Vignesh Shetty<\/a> to learn how his team is using ML in the healthcare setting.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n To provide an example, two of the best models of 2018 were decoder-only GPT (Radford et al., 2018) and encoder-only BERT (Devlin et al., 2018). Despite GPT being 36% larger (150 million parameters to 110 parameters), it actually performed 6% worse on a natural language understanding test called GLUE (75.1 vs 79.6). Some of the most famous decoder-only models are the GPT series made by OpenAI and EleutherAI. This type of model is best used for tasks like freestyle text generation and language modeling.<\/p>\n This type of model combines the encoder-only and decoder-only setups to try and get the best of both worlds. The model starts by taking an input sequence and passing it through an encoder. This encoder outputs what we call a context vector \u2014 a set of numbers of fixed size that basically represents the model\u2019s understanding of the input (think of this as the state of your brain\u2019s neurons as you are listening to someone\u2019s request). The decoder then takes this context vector and uses it to output the result.<\/p>\n This setup ends up preserving the decoder\u2019s ability to have variable length generations while reducing the inefficiencies associated with only having partial visibility of the sequence. This type of model works quite well for machine translation (e.g., translating from Language A to Language B, where a language could be a human language, a programming language, etc.), summarization, and question answering. Examples of models using this architecture include T5 (Raffel et al., 2019), BART (Liu et al., 2019), PEGASUS (Zhang et al., 2019), and Meena (Adiwardana et al., 2020).<\/p>\n<\/figure>Decoder-Only<\/strong><\/h2>\n
<\/figure>Encoder-Decoder Model<\/strong><\/h2>\n
<\/figure>