{"id":2234,"date":"2020-11-09T15:37:49","date_gmt":"2020-11-09T23:37:49","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/blog\/comet-hugging-face-integration\/"},"modified":"2020-11-09T15:37:49","modified_gmt":"2020-11-09T23:37:49","slug":"comet-hugging-face-integration","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/","title":{"rendered":"Comet \u2764\ufe0f Hugging Face"},"content":{"rendered":"\n

Hugging Face provides awesome APIs for Natural Language Modeling. In particular, they make working with large transformer models incredibly easy. These models can be used off-the-shelf for text generation, translation, and question answering, as well as downstream tasks such as text classification. The API also allows for seamless interoperability between Tensorflow and Pytorch models.\u00a0In this post we\u2019re going to show you how to get started with auto-logging model metrics and parameters to Comet from the Hugging Face transformers<\/a> library. All code for this post can be found in this Colab Notebook<\/a>.<\/p>\n\n\n\n

See the logged experiments in the Comet UI here<\/a>.<\/p>\n\n\n\n

Auto-Logging to Comet<\/h2>\n\n\n\n

To get started, you just need to have Comet installed and your Comet API key defined. You can find instructions to do that here<\/a>. Comet integrates with Hugging Face\u2019s Trainer\/TFTrainer<\/a> object, and automatically logs training metrics and parameters without requiring any changes to your source code. In order to enable logging to Comet, all we have to do is set the following environment variables.<\/p>\n\n\n\n

export COMET_API_KEY=\"YOUR API KEY\"\nexport COMET_PROJECT_NAME=\"YOUR PROJECT NAME\" <\/pre>\n\n\n\n
That\u2019s it! Auto-logging the model metrics and parameters will work as long as you use either the Trainer or TFTrainer object to train your model. Feel free to try it by running the relevant examples provided in the Hugging Face repository<\/a>.\u00a0<\/p>\n\n\n\n
Let\u2019s look at some examples of what Comet is auto-logging. We’re going to train a model to assign conference labels to research papers based on the title of the paper. We will perform a simple grid search over the batch size, learning rate and weight decay parameters for different versions of the BERT model.<\/p>\n\n\n\n
Consider the following snippet. For the sake of brevity, the data preparation steps have been omitted. You can find the full source code in this Colab Notebook<\/a>. Logged experiments from the notebook can be found here<\/a>.\u00a0<\/p>\n\n\n\n
import itertools\nimport torch\nimport transformers\nfrom transformers import AutoTokenizer\nfrom transformers import BertForSequenceClassification, Trainer, TrainingArguments\n\nfrom tqdm import tqdm\n\nPRE_TRAINED_MODEL_NAME = \"distilbert-base-uncased\"\nEPOCHS = 5\nWEIGHT_DECAY = 0.99\n\nmodel = BertForSequenceClassification.from_pretrained(\n   PRE_TRAINED_MODEL_NAME,\n   num_labels=2,\n   output_attentions=False,\n   output_hidden_states=False,\n)\n\n# Tell pytorch to run this model on the GPU.\nmodel.cuda()\n\ndecays = [0.0, 0.5, 0.99]\nlearning_rates = [5.0e-5, 3.0e-5, 2.0e-5, 1.0e-5]\nbatch_sizes = [32, 64, 128]\n\nparameters = [\n  {\"weight_decay\": x[0], \"learning_rate\": x[1], \"batch_size\": x[2]} for x in list(itertools.product(*[decays, learning_rates, batch_sizes]))\n]\n\nfor idx, p in tqdm(enumerate(parameters)):\n  weight_decay = p[\"weight_decay\"]\n  learning_rate = p[\"learning_rate\"]\n  batch_size = p[\"batch_size\"]\n\n  training_args = TrainingArguments(\n    seed=42,\n    output_dir='.\/results',\n    overwrite_output_dir=True,\n    num_train_epochs=EPOCHS,\n    per_device_train_batch_size=batch_size,\n    per_device_eval_batch_size=batch_size,\n    warmup_steps=500,\n    weight_decay=weight_decay,\n    learning_rate=learning_rate,\n    evaluation_strategy=\"epoch\",\n    do_train=True,\n    do_eval=True\n  )\n  trainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    eval_dataset=test_dataset,\n    compute_metrics=compute_metrics,\n  )\n  trainer.train()\n<\/pre>\n\n\n\n
Calling the train<\/code> method will automatically create a Comet project to log your experiment results. If you pass a callback function to compute metrics on the validation dataset, Comet will automatically log the metrics that were defined in the callback, and append the appropriate context to the name. In our example, we are computing the loss, accuracy, and f1 score on the validation set. Logging to Comet happens in real time, so we can visualize how these metrics change over the course of training.\u00a0<\/p>\n\n\n\n
\"\"\n


Figure 1. Plots for autologged metrics (<\/strong>link to experiment<\/a>)<\/strong><\/figcaption>\n<\/figure>\n\n\n\n
Comet will also automatically log all parameters defined in the TrainingArgs object, and allows searching for parameters of interest within the experiment.\u00a0<\/p>\n\n\n\n
\"\"\n

Figure 2. Logged Parameters from TrainingArgs (<\/strong>link to experiment<\/a>)<\/strong><\/figcaption>\n<\/figure>\n\n\n\n
We can log similar metrics for other versions of the BERT model by simply changing the\u00a0 PRE_TRAINED_MODEL_NAME<\/code> in the code and rerunning the Colab Notebook. A full list of model names has been provided by Hugging Face here<\/a>.<\/p>\n\n\n\n
Comet makes it easy to compare the differences in parameters and metrics between the two models<\/p>\n\n\n\n
\"\"\n

Figure 3. Differences between Two Models (<\/strong>link to experiment<\/a>)<\/strong><\/figcaption>\n<\/figure>\n\n\n\n
Additionally, we can log more advanced features such as Confusion Matrices, using the compute_metrics<\/code> function.<\/p>\n\n\n\n
from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n\ndef compute_metrics(pred):\n    experiment = comet_ml.config.get_global_experiment()\n\n    labels = pred.label_ids\n    preds = pred.predictions.argmax(-1)\n    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')\n    acc = accuracy_score(labels, preds)\n\n    experiment.log_confusion_matrix(preds, labels)\n\n    return {\n        'accuracy': acc,\n        'f1': f1,\n        'precision': precision,\n        'recall': recall\n    }<\/code><\/pre>\n\n\n\n
We can now visually inspect our precision and recall metrics, as well as view the misclassified examples.<\/p>\n\n\n\n
\"\"\n
Figure 4. Confusion Matrix of Model Predictions<\/figcaption>\n<\/figure>\n\n\n\n
Finally, we can compare the model performances across the different model parameters using a Parallel Coordinates chart.<\/p>\n\n\n\n
\"\"\n

Figure 5. Parallel Coordinates Charts of Model Hyperparameters<\/strong><\/figcaption>\n<\/figure>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
Debugging transformer models has never been easier with Comet\u2019s auto-logging integration for Hugging Face.\u00a0<\/p>\n\n\n\n
Have a look at the experiment results here<\/a> and try out the Colab Notebook here<\/a>.<\/p>\n\n\n\n
Additionally, we provide finer controls for logging through other environment variables. You can control whether you want to log to the Web UI by either setting COMET_MODE<\/code> to ONLINE<\/code> or OFFLINE<\/code>. You can also disable logging to Comet entirely by setting COMET_MODE<\/code> to DISABLE<\/code>. \u00a0<\/p>\n\n\n\n
A more comprehensive list of these environment variables is available here<\/a>. There are several experiment parameters<\/a> that can be configured through these variables and they can be set in a variety of ways.<\/a><\/p>\n\n\n\n
We hope to see transformative things being built with Comet and Hugging Face!<\/p>\n\n\n
\n\n\n
Want to stay in the loop?\u00a0Subscribe to the Comet Newsletter<\/a>\u00a0for weekly insights and perspective on the latest ML news, projects, and more.<\/em><\/h2>\n","protected":false},"excerpt":{"rendered":"
Get started with auto-logging model metrics and parameters to Comet from the Hugging Face transformers library.<\/p>\n","protected":false},"author":1,"featured_media":2240,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[8,5],"tags":[],"coauthors":[128],"class_list":["post-2234","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-comet-community-hub","category-partners-integrations"],"yoast_head":"\nComet \u2764\ufe0f Hugging Face - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Comet \u2764\ufe0f Hugging Face\" \/>\n<meta property=\"og:description\" content=\"Get started with auto-logging model metrics and parameters to Comet from the Hugging Face transformers library.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2020-11-09T23:37:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Screen-Shot-2020-11-03-at-4.39.30-PM-1024x595-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"595\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Dhruv Nair\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dhruv Nair\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Comet \u2764\ufe0f Hugging Face - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/","og_locale":"en_US","og_type":"article","og_title":"Comet \u2764\ufe0f Hugging Face","og_description":"Get started with auto-logging model metrics and parameters to Comet from the Hugging Face transformers library.","og_url":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2020-11-09T23:37:49+00:00","og_image":[{"width":1024,"height":595,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Screen-Shot-2020-11-03-at-4.39.30-PM-1024x595-1.png","type":"image\/png"}],"author":"Dhruv Nair","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Dhruv Nair","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/"},"author":{"name":"engineering@atre.net","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/550ac35e8e821db8064c5bd1f0a04e6b"},"headline":"Comet \u2764\ufe0f Hugging Face","datePublished":"2020-11-09T23:37:49+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/"},"wordCount":691,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Screen-Shot-2020-11-03-at-4.39.30-PM-1024x595-1.png","articleSection":["Comet Community Hub","Partners & Integrations"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/","url":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/","name":"Comet \u2764\ufe0f Hugging Face - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Screen-Shot-2020-11-03-at-4.39.30-PM-1024x595-1.png","datePublished":"2020-11-09T23:37:49+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Screen-Shot-2020-11-03-at-4.39.30-PM-1024x595-1.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Screen-Shot-2020-11-03-at-4.39.30-PM-1024x595-1.png","width":1024,"height":595,"caption":"Parallel Coordinates Charts of Model Hyperparameters"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/comet-hugging-face-integration\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Comet \u2764\ufe0f Hugging Face"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/550ac35e8e821db8064c5bd1f0a04e6b","name":"engineering@atre.net","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/027c18177377edf459980f0cfb83706c","url":"https:\/\/secure.gravatar.com\/avatar\/d002a459a297e0d1779329318029aee19868c312b3e1f3c9ec9b3e3add2740de?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d002a459a297e0d1779329318029aee19868c312b3e1f3c9ec9b3e3add2740de?s=96&d=mm&r=g","caption":"engineering@atre.net"},"sameAs":["https:\/\/live-cometml.pantheonsite.io"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/engineeringatre-net\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/2234","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=2234"}],"version-history":[{"count":0,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/2234\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/2240"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=2234"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=2234"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=2234"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=2234"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}