Text generation is the task of producing new text. An example of text generation using machine learning is GitHub\u2019s Copilot<\/a>, which can generate code. Apart from code generation, text generation models can:<\/p>\n Text generation can be achieved using causal language models such as GPT-2. This article will look at how we can fine-tune a causal model with custom data to generate text given a few words.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Given a sequence of tokens, Causal Language Modeling<\/strong> is the task of generating the next token. It differs from Masked Language Modeling,<\/strong> where certain words in a sentence are masked, and the model is trained to predict them.<\/p>\n In Causal Language Modeling,<\/strong> the model only considers words to the left, while Masked Language Modeling<\/strong> considers words to the left and right. Therefore, Causal Language Modeling is unidirectional,<\/strong> while Masked Language Modeling is bidirectional<\/strong>.<\/p>\n GPT<\/strong> is an example of a pre-trained Causal Language Model, while BERT<\/strong> is an example of a Masked Language Model.<\/p>\n In this example, we\u2019ll train a Causal Language Model on the wikitext using GPT. Let\u2019s start by setting up an experiment to track the training process.<\/p>\n Set up a Comet<\/a> project to track our experiments.<\/p>\n Next, log model parameters to Comet.<\/p>\n The text dataset we will be using needs to be converted into a numerical representation. It needs to be converted to a representation that the GPT model expects. Let\u2019s define a function that will tokenize the data using the Next, define a function that will group the text data and return it in small chunks that are easier to train on as they\u2019ll consume less memory.<\/p>\n Training the model with TensorFlow<\/a> requires that the dataset is converted to TensorFlow format. This is done in the following steps:<\/p>\n When you print the tokenized dataset you will see:<\/p>\n Given some sentences are longer than others, we have to ensure that they are of the same length. In this case, the model’s maximum length is 1024.<\/p>\n To ensure that all sentences have the same length, we have to truncate longer sentences and pad shorter ones using a padding token<\/strong>. Therefore, we need a way to inform the model to ignore the padding tokens. This is done using an attention mask<\/strong>. The attention mask is a tensor with the same shape as the Comet is completely free for academics! Sign up today<\/a> and get started with just two lines of code.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n With the dataset in the correct format, you can now define the causal language model in TensorFlow<\/a>. The definition of the causal model is done by instantiating Since this is a pre-trained model, we train it at a low learning rate to avoid overfitting. Training is done by calling the Once training is complete, we evaluate the model on the validation set and log the loss. Causal language models are evaluated using the cross-entropy loss and the perplexity. In this case, we use perplexity. The perplexity is the exponential of the cross-entropy.<\/p>\n Comet will log all TensorFlow metrics and hyperparameters automatically.<\/p>\n Comet also creates the histograms of the training process. From these visuals, you can see:<\/p>\n Let\u2019s now try and generate some tokens from the trained causal model. This is done by:<\/p>\n We also log the input and generated sentences to Comet and eventually end the experiment.<\/p>\n In this article, you have seen how to create a text generation model using Hugging Face and Comet. You can tweak the project by trying other types of language models.<\/p>\n Follow me on LinkedIn<\/a> for more technical resources.<\/p>\n\n
Causal Language Modeling vs. Masked Language Modeling<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:383\/0*4qYkQYogkls9hbxY.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:382\/0*ZyMpLDw1P7ulg4i0.png\")
<\/figure>Getting started<\/h2>\n
import comet_ml\n\nexperiment = comet_ml.Experiment(\n api_key=\"YOUR_API_KEY\",\n project_name=\"clm\", log_code=True,\n auto_metric_logging=True,\n auto_param_logging=True,\n auto_histogram_weight_logging=True,\n auto_histogram_gradient_logging=True,\n auto_histogram_activation_logging=True,\n)<\/pre>\n
import comet_ml\n\nexperiment = comet_ml.Experiment(\n api_key=\"your_API_KEY\",\n project_name=\"clm\", log_code=True,\n auto_metric_logging=True,\n auto_param_logging=True,\n auto_histogram_weight_logging=True,\n auto_histogram_gradient_logging=True,\n auto_histogram_activation_logging=True,\n)<\/pre>\n
# these will all get logged\nparams = {\n \"model\": \"gpt2\",\n \"epochs\": 50,\n \"batch_size\": 32,\n \"learning_rate\": 2e-5,\n \"weight_decay\": 0.01,\n\n}\n\nexperiment.log_parameters(params)<\/pre>\nTokenize dataset<\/h2>\n
GPT2Tokenizer<\/code>.<\/p>\ndef tokenize_function(examples):\n from transformers import GPT2Tokenizer\n tokenizer = GPT2Tokenizer.from_pretrained(params['model'])\n return tokenizer(examples[\"text\"])<\/pre>\n
def group_texts(examples):\n # Concatenate all texts.\n concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n total_length = len(concatenated_examples[list(examples.keys())[0]])\n # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n # customize this part to your needs.\n # block_size = tokenizer.model_max_length\n block_size = 128\n total_length = (total_length \/\/ block_size) * block_size\n # Split by chunks of max_len.\n result = {\n k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n for k, t in concatenated_examples.items()\n }\n result[\"labels\"] = result[\"input_ids\"].copy()\n return result<\/pre>\nConvert data to TensorFlow format<\/h2>\n
\n
group_texts<\/code> function.<\/li>\nto_tf_dataset<\/code>function.<\/li>\n<\/ul>\nfrom datasets import load_dataset\nfrom transformers import DefaultDataCollator\n\ndatasets = load_dataset(\"wikitext\", \"wikitext-2-raw-v1\")\ntokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=[\"text\"])\nlm_datasets = tokenized_datasets.map(\n group_texts,\n batched=True,\n batch_size=1000,\n num_proc=4,)\ndata_collator = DefaultDataCollator(return_tensors=\"tf\")\ntrain_set = lm_datasets[\"train\"].to_tf_dataset(\n columns=[\"attention_mask\", \"input_ids\", \"labels\"],\n shuffle=True,\n batch_size=params['batch_size'],\n collate_fn=data_collator,)\n\nvalidation_set = lm_datasets[\"validation\"].to_tf_dataset(\n columns=[\"attention_mask\", \"input_ids\", \"labels\"],\n shuffle=False,\n batch_size=params['batch_size'],\n collate_fn=data_collator,)<\/pre>\n
\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*dOi052ZH3Fzz7bBaarrWdQ.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*WN-AdACRLc5qZaXWgHrVRA.png\")
<\/figure>input_ids<\/code>. The input_ids<\/code> are the numerical representation of the sentences. The attention mask contains zeros and ones indicating whether the tokens should be attended to. As a result, the padding tokens will be ignored.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\nDefine causal language model<\/h2>\n
TFAutoModelForCausalLM<\/code> and passing the desired config. In this case, it\u2019s a pre-trained GPT2 model.<\/p>\nfrom transformers import AutoTokenizer\nfrom transformers import AutoConfig, TFAutoModelForCausalLM\nfrom transformers import AdamWeightDecay\nmodel_checkpoint = params['model']\nconfig = AutoConfig.from_pretrained(model_checkpoint)\ngpt2 = TFAutoModelForCausalLM.from_config(config)\n<\/pre>\n
Train Causal language model<\/h2>\n
fit<\/code> method while passing the training and validation set.<\/p>\nimport math\n\nlearning_rate = params['learning_rate']\nweight_decay = params['weight_decay']\noptimizer = AdamWeightDecay(learning_rate=learning_rate, weight_decay_rate=weight_decay)\ngpt2.compile(optimizer=optimizer)\ngpt2.fit(train_set, validation_data=validation_set, epochs=params['epochs'])\neval_loss = gpt2.evaluate(validation_set)\nexperiment.log_metrics({\"eval_loss\":eval_loss})\nprint(f\"Perplexity: {math.exp(eval_loss):.2f}\")<\/pre>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*V6BYHoChHkWJxUSzkLWHZA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*cGybQmCCFGfxeT4wCYfQ_A.png\")
<\/figure>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*aXOEkPUKUQRy63bobJ6RFQ.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*A0qvhL2JYobmuGq35aCIsg.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*l0QUSfwYcFvZjtT2akM2EQ.png\")
<\/figure>Test causal language model<\/h2>\n
\n
generate<\/code> method while passing the tokenized sentence.<\/li>\ninput_sequence = \"The ship\"\nexperiment.log_text(input_sequence)\ntokenizer_checkpoint = params['model']\ntokenizer = AutoTokenizer.from_pretrained(tokenizer_checkpoint)\n# encode context the generation is conditioned on\ninput_ids = tokenizer.encode(input_sequence, return_tensors='tf')\noutput = gpt2.generate(input_ids,min_length=20)\nprint(\"Output:\\n\" + 100 * '-')\nresult = tokenizer.decode(output[0], skip_special_tokens = True)\nexperiment.log_text(result)\nexperiment.end()\nresult<\/pre>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*dTZjKXQmOC7VapM3rruWsA.png\")
<\/figure>Final thoughts<\/h2>\n
Resources<\/h2>\n