If you are a natural language processing researcher or have an interest in this field, you have surely come across SpaCy<\/a> or you are very close to it!<\/p>\n SpaCy, a free, open source natural language processing library developed in Python, is very popular for use in real products. This advanced library, which is frequently used in applications that process and understand large volumes of text, is now integrated with Comet ML<\/a>, a very useful experiment monitoring tool!<\/p>\n Now let\u2019s train a multi-label text classifier using SpaCy-v3 on the Huggingface \u2014 dair-ai\/emotion<\/strong><\/a> dataset and track the model trainings and record the results with Comet ML!<\/p>\n The dataset to be used in the application described in the blog post was created for an emotion classification task. For detailed information on the dataset of Twitter messages written in English, which includes six basic emotions: anger, fear, joy, love, sadness and surprise,<\/strong> please review this article.<\/a><\/p>\n \ud83d\udd39Please click<\/strong><\/a> to access the Emotion dataset via Hugging Face Datasets!<\/p> \ud83d\udd39Please click<\/strong><\/a> to download the Emotion dataset directly!<\/p> \ud83d\udd39Please click<\/strong><\/a> to access Papers with Code Public Leaderboard!<\/p><\/blockquote>\n \ud83d\udd38Data fields are:<\/p>\n \ud83d\udd38An example within the dataset looks like this:<\/p>\n \ud83d\udd38A total of 20,000 samples in the dataset are divided into training, validation, and test sets:<\/p>\n The first step is to install the necessary libraries, including SpaCy, which we will use for basic language processing tasks, Comet ML, which we will use to log the model trainings and results, and HuggingFace\u2019s datasets library for accessing the dataset.<\/p>\n To configure Comet, you can set your credentials via environment variables as follows:<\/p>\n \u201cDatasets\u201d <\/strong>is a library that allows loading datasets that can be used for audio processing, image processing, natural language processing tasks, and more, with a single line of code.<\/p>\n Let\u2019s load the \u2018Emotions\u2019 dataset using this library and create a smaller subset of the dataset\u2019s training, validation and test sets as follows.<\/p>\n Let\u2019s assign all the labels in the dataset as an array to a variable called \u201ccategories.\u201d<\/strong><\/p>\n Let\u2019s define our labels in dictionary format as \u201ctag name\u201d<\/strong> and \u201ctag id\u201d <\/strong>to be used when defining in SpaCy data format.<\/p>\n We need to convert the text and tags to clean SpaCy Doc Objects.<\/p>\n Let\u2019s convert the train, validation, and test datasets to the SpaCy Doc object type and save it to disk with the name we gave it.<\/p>\n Sometimes simple solutions offer the best results. We made minor hardware optimizations for a huge increase in throughput. Check out the project here<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n The training in SpaCy v3 needs a config file to set up your model and all the hyperparameters. Let\u2019s use SpaCy\u2019s built-in tools to create a configuration. You can create a configuration with the following command:<\/p>\n Let\u2019s make the following changes in the created configuration file to activate the Comet ML experiment tracking tool and keep the training logs in Comet ML.<\/p>\n For detailed information on the integration of SpaCy with Comet ML, you can review the official documentation<\/strong><\/a>.<\/strong><\/p>\n [comet]<\/p>\n\n\n\n api_key=<COMET API KEY>\nproject_name=<COMET PROJECT NAME>\n<\/p>\n\n\n\n If you have completed the previous 6 steps, you are now ready to train a model!<\/p>\n\n\n\n By giving the config file and training dataset we prepared to the \u201cspacy_train\u201d function, we can start the training as follows:<\/p>\n\n\n\n During the training period, you can see the training logs instantly under the workspace project on Comet.<\/p>\n\n\n\n You can also see the training outputs in the notebook.<\/p>\n\n\n\n When the experiment is complete, two models are created in the \ud83d\udd38model-best: <\/strong>model with highest score during training iteration<\/p>\n\n\n\n \ud83d\udd38model-last:<\/strong> model trained in last training iteration<\/p>\n\n\n\n Evaluation of the model performance with the test dataset that did not participate in the training can be done as follows:<\/p>\n\n\n\n Performance results in the test set of the model:<\/p>\n\n\n\n We can quickly use our trained classifier model to get predictions about the sentiment of different texts.<\/p>\n\n\n\n Let\u2019s load our best trained SpaCy model from disk and then easily get the sensitivity analysis model result for all sample sentences with the SpaCy Pipeline. Finally, let\u2019s filter the scores above the 0.5 threshold value among the multiple classes of the model and print the results.<\/p>\n\n\n\n To access all the code in this emotion classification application example, please review the Colab Notebook<\/a>!<\/p>\n\n\n\n Today, data science researchers use many machine learning frameworks and third-party tools in their work. Comet integrates with the most commonly used platforms and tools, and its modular and customizable design allows researchers to maintain flexibility to allow their machine learning platform to adapt to future changes and needs.<\/p>\n Please visit the official Comet documentation page<\/strong><\/a> to review other platforms and tools that Comet integrates with!<\/p>\nEmotion Recognition Dataset Overview<\/h1>\n
\n
text<\/code>: a string feature.<\/li>\nlabel<\/code>: a classification label, with possible values including sadness (0), joy (1), love (2), anger (3), fear (4), surprise (5)<\/strong><\/li>\n<\/ul>\n{<\/span>\n \"text\"<\/span>:<\/span> \"im feeling quite sad and sorry for myself but ill snap out of it soon\"<\/span>,<\/span>\n \"label\"<\/span>:<\/span> 0<\/span>\n}<\/span><\/span><\/pre>\n\n
Step 1 : Setup<\/h2>\n
!pip install \"spacy >= 3.0.6\"<\/span>\n!pip install \"comet_ml>=3.31.19\"<\/span>\n!pip install datasets<\/span><\/pre>\nStep 2: Comet Environment Variables<\/h2>\n
%env COMET_API_KEY=<Your Comet API Key>\n%env COMET_PROJECT_NAME=<Your Comet Project Name><\/span><\/pre>\nStep 3: Loading Dataset Using Huggingface Datasets<\/h2>\n
small_train_dataset = ds[\"train\"<\/span>].shuffle(seed=34<\/span>).take(5000<\/span>)\nsmall_val_dataset = ds[\"validation\"<\/span>].shuffle(seed=34<\/span>).take(1000<\/span>)\nsmall_test_dataset = ds[\"test\"<\/span>].shuffle(seed=34<\/span>).take(1000<\/span>)<\/span><\/pre>\ncategories = small_train_dataset.features[\"label\"<\/span>].names<\/span><\/pre>\nlabels_ = {}\nfor<\/span> index, key in<\/span> enumerate<\/span>(categories):\n labels_[key] = index<\/span><\/pre>\nStep 4: Convert to spaCy Data Format and Save to Disk<\/h2>\n
def<\/span> convert_spacy_dataset<\/span>(dataset, target_file: str<\/span>, labels<\/span>):\n nlp = spacy.blank(\"en\"<\/span>)\n db = DocBin()\n\n for<\/span> item in<\/span> tqdm(dataset):\n doc = nlp.make_doc(item[\"text\"<\/span>])\n doc.cats = {label: 0<\/span> for<\/span> label in<\/span> labels}\n doc.cats[labels[item[\"label\"<\/span>]]] = 1<\/span>\n\n db.add(doc)\n\n db.to_disk(target_file)\n return<\/span> db<\/span><\/pre>\nconvert_spacy_dataset(small_test_dataset, \"test_data.spacy\"<\/span>, categories)\nconvert_spacy_dataset(small_val_dataset, \"validation_data.spacy\"<\/span>, categories)\nconvert_spacy_dataset(small_train_dataset, \"train_data.spacy\"<\/span>, categories)<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\nStep 5: Create the Configuration File<\/h2>\n
python -m spacy init config --lang en --pipeline textcat_multilabel comet_config.cfg<\/span><\/pre>\nStep 6: Integration with Comet<\/h2>\n
[training.logger]\n@loggers = \"comet_ml.spacy.logger.v1\"\nworkspace=\"<COMET WORKSPACE NAME>\"\nproject_name=<COMET PROJECT NAME>\nremove_config_values = [\"paths.train\", \"paths.dev\", \"corpora.train.path\", \"corpora.dev.path\"]<\/span><\/pre><\/div><\/div><\/div>\n\n\n![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*HmYSNRYP7B_EAU9HgRb4pg.png\")
Step 7: Model Training and Evaluation<\/h2>\n\n\n\n
from<\/span> spacy.cli.train import<\/span> train as<\/span> spacy_train\n\nconfig_path = \"comet_config.cfg\"<\/span>\noutput_model_path = \"output_models\/\"<\/span>\n\nspacy_train(\n config_path,\n output_path=output_model_path,\n overrides={\n \"paths.train\"<\/span>: \"train_data.spacy\"<\/span>,\n \"paths.dev\"<\/span>: \"validation_data.spacy\"<\/span>,\n },\n use_gpu=0<\/span>\n)<\/span><\/pre>\n\n\n\n![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*V6z0bT81TChVt0P1nYmxrg.png\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*QelXNMTMYpMNn8PwgzbnQQ.png\")
<\/figure>\n\n\n\noutput_models<\/code> folder:<\/p>\n\n\n\npython -m spacy evaluate output_models\/model-best\/ test_data.spacy<\/span><\/pre>\n\n\n\n================================== Results ==================================\n\nTOK 100.00\nTEXTCAT (macro AUC) 94.46\nSPEED 281793\n\n=========================== Textcat F (per label) ===========================\n\n P R F\nsadness 91.42 75.80 82.88\njoy 90.88 83.53 87.05\nlove 76.00 44.71 56.30\nanger 93.83 58.91 72.38\nfear 94.94 61.48 74.63\nsurprise 80.00 21.62 34.04\n\n======================== Textcat ROC AUC (per label) ========================\n\n ROC AUC\nsadness 0.96\njoy 0.96\nlove 0.94\nanger 0.96\nfear 0.94\nsurprise 0.91<\/span><\/pre>\n\n\n\nStep 8: Using the Trained Model<\/h2>\n\n\n\n
#We load our spaCy model from disk<\/span>\ntrained_nlp = spacy.load(\"output_models\/model-best\/\"<\/span>)\n\ntexts = ['im feeling pretty anxious'<\/span>,\n 'i don t feel particularly agitated'<\/span>,\n 'i find myself in the odd position of feeling supportive of'<\/span>,\n 'i feel so cold a href http irish'<\/span>,\n 'im feeling very peaceful about our wedding again now after having'<\/span>\n ]\n\ncategory_scores = [doc.cats for<\/span> doc in<\/span> trained_nlp.pipe(texts)]\n\nthresh = 0.5<\/span>\nfor<\/span> d in<\/span> category_scores:\n print<\/span>(dict<\/span>((k, v) for<\/span> k, v in<\/span> d.items() if<\/span> v >= thresh))<\/span><\/pre>\n\n\n\n{'fear': 0.6161302924156189}\n{'anger': 0.7113955020904541}\n{'love': 0.5419280529022217}\n{'anger': 0.8137673139572144}\n{'joy': 0.6693308353424072}<\/span><\/pre>\n\n\n\n