BERT \u2014 Bidirectional Encoder Representations from Transformers \u2014 is a pre-trained language model for natural language processing tasks such as text classification and question and answering. This article will look at fine-tuning the BERT for text classification. In the end, the BERT model will learn to label if a review from the To understand how the model is learning, we need to visualize histograms of the weights and biases, the activations and gradients. To achieve that, we use Comet to track the project. Comet<\/a> automatically tracks these and other items such as:<\/p>\n When using Comet, these items are logged by default, but you can manually configure what will be logged.<\/p>\n Logging various parameters makes it easy to update them and compare how they affect the model’s performance. You can easily change a parameter when all parameters are saved in one dictionary. The We\u2019ll use the Next, apply the function to the dataset. The map function applies the tokenization function to all the sentences. Next, shuffle the data and select the number of data points you would like to use.<\/p>\n We\u2019ll fine-tune the BERT model in TensorFlow<\/a>. Let\u2019s convert the dataset to a TensorFlow dataset format. Hugging Face provides the The The Innovation and academia go hand-in-hand. Listen to our own CEO Gideon Mendels chat with the Stanford MLSys Seminar Series team<\/a> about the future of MLOps and give the Comet platform<\/a> a try for free!<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Since auto-logging is active, you will see live results of the model training on Comet. On the charts panel, you will see graphs for the:<\/p>\n The Code<\/strong> tab will show the code used in this experiment. On the hyperparameters tab, you will see all the logged parameters.<\/p>\n All model metrics can be viewed from the Metrics<\/strong> tab.<\/p>\n Click the System Metrics<\/strong> tab to see the Memory Usage and CPU Utilization for the model training process.<\/p>\n Click the Histograms<\/strong> tab to see histograms for the weights and biases, activations, and gradients.<\/p>\n Check how the BERT model performs on new data. You can also log the test sentence to Comet. First, tokenize the input data, then pass it to the BERT model. It will output logits which you will need to decode.<\/p>\n Let\u2019s interpret the prediction and log it as well. You can get the predicted class by passing the logits to End the experiment to make sure all items are logged as expected.<\/p>\n This article has shown you how to fine-tune a BERT model for text classification while tracking the model using Comet<\/a>. You can improve this model by increasing the amount of training data. You can also swap the BERT model with another Hugging Face transformer<\/a> model and compare the performance.<\/p>\n Follow me on LinkedIn<\/a> for more technical resources.<\/p>\nimdb<\/code> dataset is positive or negative.<\/p>\n\n
Getting started<\/h2>\n
import comet_ml\n\nexperiment = comet_ml.Experiment(\n api_key=\"YOUR_API_KEY\",\n project_name=\"HF\", log_code=True,\n auto_metric_logging=True,\n auto_param_logging=True,\n auto_histogram_weight_logging=True,\n auto_histogram_gradient_logging=True,\n auto_histogram_activation_logging=True,\n)<\/pre>\n
Log parameters<\/h2>\n
log_parameters<\/code>function is used for logging a dictionary of parameters in Comet.<\/p>\n# these will all get logged\nparams = {\n \"bert\": \"bert-base-uncased\",\n \"num_labels\": 2,\n \"return_tensors\": \"tf\",\n \"batch_size\": 8,\n \"epochs\": 3,\n \"padding\":\"max_length\",\n \"truncation\": True,\n \"dataset\": \"imdb\",\n}\n\nexperiment.log_parameters(params)<\/pre>\nTokenize text data<\/h2>\n
imdb<\/a><\/code> dataset<\/a> to fine-tune BERT. Create a numerical representation of the data because it\u2019s in text form. Use the BertTokenizer<\/code>since you are fine-tuning a BERT model. This ensures that the data is in the form that the BERT requires. Next, we define a function that will tokenize the data and apply a maximum length and truncation to ensure that all sentences are the same length. Tokenizing the data converts it to a numerical representation that\u2019s acceptable by the machine learning model. You can\u2019t pass the raw sentences to the model.<\/p>\ndef tokenize_function(examples):\n from transformers import BertTokenizer\n tokenizer = BertTokenizer.from_pretrained(params['bert'])\n return tokenizer(examples[\"text\"], padding=params[\"padding\"], truncation=params[\"truncation\"])<\/pre>\n
from datasets import load_dataset\n\ndataset = load_dataset(params['dataset'])\n\ntokenizer = AutoTokenizer.from_pretrained(params['bert'])\ntokenized_datasets = dataset.map(tokenize_function, batched=True)\n\nsmall_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\nsmall_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))<\/pre>\n
Create TensorFlow dataset<\/h2>\n
DefaultDataCollator<\/code> function to batch the dataset and perform data augmentation. After that, use the to_tf_dataset<\/code> function to convert the dataset to TensorFlow format.<\/p>\nto_tf_dataset<\/code> method allows you to define the columns and labels included in the dataset. Converting the data to TensorFlow makes it possible to train the model using the fit<\/code> method and later evaluate it using the evaluate<\/code> method.<\/p>\nfrom transformers import DefaultDataCollator\ndata_collator = DefaultDataCollator(return_tensors=params['return_tensors'])\n\ntf_train_dataset = small_train_dataset.to_tf_dataset(\n columns=[\"attention_mask\", \"input_ids\", \"token_type_ids\"],\n label_cols=[\"labels\"],\n shuffle=True,\n collate_fn=data_collator,\n batch_size=params['batch_size'],)\n\ntf_validation_dataset = small_eval_dataset.to_tf_dataset(\n columns=[\"attention_mask\", \"input_ids\", \"token_type_ids\"],\n label_cols=[\"labels\"],\n shuffle=False,\n collate_fn=data_collator,\n batch_size=params['batch_size'],)<\/pre>\n
Train BERT model<\/h2>\n
TFAutoModelForSequenceClassification<\/code> is a model class with a sequence classification head. We can use it to initialize a pre-trained BERT classification model. Next, compile the model under a low learning rate and fit it to the data. Using a low learning rate is important in transfer learning<\/a> to ensure that we don\u2019t overfit the model.<\/p>\nimport tensorflow as tf\nfrom transformers import TFAutoModelForSequenceClassification\n\nbert = TFAutoModelForSequenceClassification.from_pretrained(params['bert'], num_labels=params['num_labels'])\nbert.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=5e-5),loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),metrics=tf.metrics.SparseCategoricalAccuracy(),)\nbert.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=params['epochs'])<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
Evaluate model performance<\/h2>\n
\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:600\/1*DoEhNrX7caigNd7r8RCYEQ.gif\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Cve29aIFsdchEuKxzHhZ2A.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*TGnD1Vdh4TxfLuQEbyI_ow.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*ENqKVpYPnc8n-aQ4ibj0sA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:600\/1*iliafQDV49Ct7OCU9ndLBQ.gif\")
<\/figure>Test model on new data<\/h2>\n
input_sequence = \"I hated that movie, it was too slow\"\nexperiment.log_text(input_sequence)\n# encode context the generation is conditioned on\ninput_ids = tokenizer.encode(input_sequence, return_tensors='tf')\noutput = bert(input_ids)\nlogits = output.logits<\/pre>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*5zCSyOGTPmDDDM5PapJezg.png\")
<\/figure>tf.math.argmax<\/code>. Passing the predicted class to bert.config.id2label<\/code> will give you the predicted label.<\/p>\npredicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])\nprediction = bert.config.id2label[predicted_class_id]\nexperiment.log_text(prediction)\nprediction<\/pre>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*hzvk4p2Ul-6-WkPRuzLFRQ.png\")
<\/figure>experiment.end()<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
Final thoughts<\/h1>\n
Resources<\/h2>\n