{"id":8023,"date":"2023-10-25T14:31:21","date_gmt":"2023-10-25T22:31:21","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=8023"},"modified":"2025-04-24T17:05:17","modified_gmt":"2025-04-24T17:05:17","slug":"sentiment-analysis-with-sparknlp-and-comet","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet\/","title":{"rendered":"Sentiment Analysis With SparkNLP and Comet"},"content":{"rendered":"\n
\n
\n
\n
\n
\n
\"A<\/figure>
<\/picture><\/div>\n<\/div>
Picture by Anna Nekrashevich<\/a>, Pexels.com<\/figcaption><\/figure>\n

Introduction<\/h1>\n

Sentiment analysis is a natural language processing technique which identifies and extracts subjective information from source materials using computational linguistics and text analysis. It is employed to ascertain a speaker\u2019s or writer\u2019s feelings, attitudes, and opinions with regard to a particular subject or general contextual polarity of a work. Applications for sentiment analysis include market research, customer service, and social media monitoring. Among other things, companies use it to get insight from their customers on social media and know what is being said about their products.<\/p>\n

Spark NLP is a natural language processing library built on Apache Spark. It is used to carry out precise annotation for machine learning pipelines that can scale quickly in a distributed environment.<\/p>\n

We will build a pipeline for performing sentiment analysis on text data using the Spark NLP library and use Comet to monitor the metrics of our model. Comet is an online platform that allows you to track and monitor logs experiments and we are using Comet in this tutorial to track the metrics of our model.<\/p>\n

But first, let\u2019s understand how sentiment analysis works.<\/p>\n

Sentiment Analysis<\/h1>\n

One common approach to sentiment analysis is using supervised machine learning techniques to classify text into various sentiment categories, such as positive, negative, or neutral. This is often done by training a classifier on a labeled dataset of text, where each piece of text has been manually annotated with a sentiment label. Once the classifier is trained, it can be used to classify new text into the appropriate sentiment category.<\/p>\n

Another approach is to use lexicon-based methods, which involve looking up words or phrases in a pre-defined lexicon to determine the sentiment. This can be done by using a lexicon of positive and negative words and counting the number of positive and negative words in the text. The sentiment of the text is then determined based on the ratio of positive to negative words.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

How does the team at Uber manage to keep their data organized and their team united? Comet\u2019s experiment tracking. Learn more from Uber\u2019s Olcay Cirit<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Now let\u2019s begin by installing the necessary packages:<\/span><\/div>\n\n\n\n
\n
\n
\n
\u200b\u200bpip install spark-nlp==4.2.6\npip install comet_ml<\/span><\/pre>\n
\n
\n
\"A<\/figure>
<\/picture><\/div>\n<\/div>\n
Comet homepage<\/figcaption>\n<\/figure>\n
If you don\u2019t already have a Comet account, you can create one here<\/a>. Once your account is set up, you can manually create a project from the UI, or, as you\u2019ll see in this tutorial, we can create our project with a simple experiment parameter.<\/p>\n
\n
\n
\"A<\/figure>
<\/picture><\/div>\n<\/div>\n
Creating a new Comet project manually in the UI<\/figcaption>\n<\/figure>\n
After creating an account, navigate to the top right of the Comet website and click on your avatar. Then select Account Settings and finally pick the API keys tab and generate an API key.<\/p>\n
\n
\n
\"A<\/figure>
<\/picture><\/div>\n<\/div>\n
Getting API keys<\/figcaption>\n<\/figure>\n
Now let\u2019s import our libraries:<\/p>\n
# Import Spark NLP<\/span>\nimport<\/span> sparknlp\nfrom<\/span> sparknlp.base import<\/span> *\nfrom<\/span> sparknlp.annotator import<\/span> *\nfrom<\/span> sparknlp.pretrained import<\/span> PretrainedPipeline\n\n\nfrom<\/span> pyspark.ml import<\/span> Pipeline\nfrom<\/span> pyspark.sql import<\/span> SparkSession\nimport<\/span> pyspark.sql.functions as<\/span> F\n\nspark = sparknlp.start()\n\n# Import comet<\/span>\nimport<\/span> comet_ml\nfrom<\/span> sparknlp.logging.comet import<\/span> CometLogger<\/span><\/pre>\n
Note that Comet integrates with Spark automatically<\/a>. Next, we instantiate our experiment object:<\/p>\n
experiment = comet_ml.Experiment(\"YOUR-API-KEY\"<\/span>)<\/span><\/pre>\n
Now we\u2019ll give our project a name using the project_name<\/code> parameter:<\/p>\n
comet_ml.login(project_name='test-sparknlp '<\/span>)<\/span><\/pre>\n
Next, we start Spark, set the logger, and set an output log path:<\/p>\n
spark = sparknlp.start()\nlogger = CometLogger()\nOUTPUT_LOG_PATH = \".\/run\"<\/span><\/span><\/pre>\n
Building the pipeline<\/h1>\n
We have five stages in our pipeline:<\/p>\n
    \n
  1. The first stage is the DocumentAssembler<\/code>, which creates a column from the text that is imputed. This allows us to easily use the text when we want to train the model.<\/li>\n<\/ol>\n
    # Create a document assembler<\/span>\ndocument = DocumentAssembler() \\\n    .setInputCol(\"text\"<\/span>) \\\n    .setOutputCol(\"document\"<\/span>)<\/span><\/pre>\n
    2. The second stage is the Tokenizer<\/code>, which tokenizes the text from the document<\/code> column into separate words in the output as token<\/code>. We don’t know what the input text will look like, so we\u2019ll have to format it to fit our guidelines by slicing sentences into words that our model can easily digest.<\/p>\n
    # Create a tokenizer<\/span>\ntoken = Tokenizer() \\\n    .setInputCols([\"document\"<\/span>]) \\\n    .setOutputCol(\"token\"<\/span>)<\/span><\/pre>\n
    3. The third step is the Normalizer<\/code>, which removes the punctuation from the token<\/code> column, makes the text lowercase, and then saves the output column as normal<\/code>.<\/p>\n
    # Create a normalizer<\/span>\nnormalizer = Normalizer() \\\n    .setInputCols([\"token\"<\/span>]) \\\n    .setOutputCol(\"normal\"<\/span>)<\/span><\/pre>\n
    4. The fourth step is instantiating the ViveknSentimentModel<\/code>. Here we\u2019re using a pre-trained model to classify the sentiment of the text as positive<\/code>, negative<\/code>, or neutral<\/code> by taking the document<\/code> and normal<\/code> columns as input and setting its output column to result_sentiment<\/code>. Now we\u2019re ready to actually train the model on our dataset.<\/p>\n
    # Load a pretrained sentiment analysis model<\/span>\nvivekn = ViveknSentimentModel.pretrained() \\\n    .setInputCols([\"document\"<\/span>, \"normal\"<\/span>]) \\\n    .setOutputCol(\"result_sentiment\"<\/span>)<\/span><\/pre>\n
    5. The fifth stage takes the output of the previous stage, result_sentiment<\/code>and sets the final output column as final_sentiment<\/code>. The aim is just to generate a final sentiment output for the user and save it as finisher.<\/p>\n
    # Create a finisher<\/span>\nfinisher = Finisher() \\\n    .setInputCols([\"result_sentiment\"<\/span>]) \\\n    .setOutputCols(\"final_sentiment\"<\/span>)\n\nlogger.monitor(logdir=OUTPUT_LOG_PATH, model=vivekn)\n\n# Create a pipeline with all the stages<\/span>\npipeline = Pipeline().setStages([document, token, normalizer, vivekn, finisher])\n<\/span><\/pre>\n
    After creating the pipeline, let\u2019s fit in a small dataset of two sentences and get the sentiment of each of them displayed.<\/p>\n
    # Create the data as a Spark dataframe<\/span>\ndata = spark.createDataFrame([\n    [\"I recommend this movie\"<\/span>],\n    [\"Dont waste your time!!!\"<\/span>]\n]).toDF(\"text\"<\/span>)\n\n# Fit the pipeline to the data<\/span>\npipelineModel = pipeline.fit(data)\n\n# Transform the data and show the final sentiment<\/span>\nresult = pipelineModel.transform(data)\nresult.select(\"final_sentiment\"<\/span>).show(truncate=False<\/span>)\nlogger.log_pipeline_parameters(pipelineModel)\nlogger.end()<\/span><\/pre>\n
    \n
    \n
    \"Screenshot<\/figure>
    <\/picture><\/div>\n<\/div>\n<\/figure>\n
    Now you can see the sentiment of the text that you passed it. You can display the evaluation metrics directly in your interactive notebook using the code below, or head over to the Comet UI to check out the results more in-depth.<\/p>\n
    \n
    \n
    \"A<\/figure>
    <\/picture><\/div>\n<\/div>\n
    Comet dashboard<\/figcaption>\n<\/figure>\n
    Conclusion<\/h1>\n
    This is how to build a sentiment analysis pipeline with SparkNLP, while also using Comet to track your project metrics. You can check out the code on this Colab notebook<\/a> to easily get started. There are many other cool ways to use Comet and you can check them out and try more examples from their documentation<\/a>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
    Picture by Anna Nekrashevich, Pexels.com Introduction Sentiment analysis is a natural language processing technique which identifies and extracts subjective information from source materials using computational linguistics and text analysis. It is employed to ascertain a speaker\u2019s or writer\u2019s feelings, attitudes, and opinions with regard to a particular subject or general contextual polarity of a work. […]<\/p>\n","protected":false},"author":100,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[9],"tags":[],"coauthors":[198],"class_list":["post-8023","post","type-post","status-publish","format-standard","hentry","category-product"],"yoast_head":"\nSentiment Analysis With SparkNLP and Comet - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Sentiment Analysis With SparkNLP and Comet\" \/>\n<meta property=\"og:description\" content=\"Picture by Anna Nekrashevich, Pexels.com Introduction Sentiment analysis is a natural language processing technique which identifies and extracts subjective information from source materials using computational linguistics and text analysis. It is employed to ascertain a speaker\u2019s or writer\u2019s feelings, attitudes, and opinions with regard to a particular subject or general contextual polarity of a work. […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-25T22:31:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:05:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*crmBmr5sXOkj44ZNzZY2fw.jpeg\" \/>\n<meta name=\"author\" content=\"Sandy M\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sandy M\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Sentiment Analysis With SparkNLP and Comet - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet","og_locale":"en_US","og_type":"article","og_title":"Sentiment Analysis With SparkNLP and Comet","og_description":"Picture by Anna Nekrashevich, Pexels.com Introduction Sentiment analysis is a natural language processing technique which identifies and extracts subjective information from source materials using computational linguistics and text analysis. It is employed to ascertain a speaker\u2019s or writer\u2019s feelings, attitudes, and opinions with regard to a particular subject or general contextual polarity of a work. […]","og_url":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-10-25T22:31:21+00:00","article_modified_time":"2025-04-24T17:05:17+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*crmBmr5sXOkj44ZNzZY2fw.jpeg","type":"","width":"","height":""}],"author":"Sandy M","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Sandy M","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet\/"},"author":{"name":"Sandy M","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/46eba004d124beb6d3592cf6728f43d6"},"headline":"Sentiment Analysis With SparkNLP and Comet","datePublished":"2023-10-25T22:31:21+00:00","dateModified":"2025-04-24T17:05:17+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet\/"},"wordCount":829,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*crmBmr5sXOkj44ZNzZY2fw.jpeg","articleSection":["Product"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet\/","url":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet","name":"Sentiment Analysis With SparkNLP and Comet - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*crmBmr5sXOkj44ZNzZY2fw.jpeg","datePublished":"2023-10-25T22:31:21+00:00","dateModified":"2025-04-24T17:05:17+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*crmBmr5sXOkj44ZNzZY2fw.jpeg","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*crmBmr5sXOkj44ZNzZY2fw.jpeg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/sentiment-analysis-with-sparknlp-and-comet#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Sentiment Analysis With SparkNLP and Comet"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/46eba004d124beb6d3592cf6728f43d6","name":"Sandy M","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/73e571000fd489681999ca76cba0070b","url":"https:\/\/secure.gravatar.com\/avatar\/b91e4581668129edd364e880f0a56403a4ab1598cbb65e62ca9348c6db10ba72?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b91e4581668129edd364e880f0a56403a4ab1598cbb65e62ca9348c6db10ba72?s=96&d=mm&r=g","caption":"Sandy M"},"url":"https:\/\/www.comet.com\/site\/blog\/author\/mondaysandy3gmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/8023","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=8023"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/8023\/revisions"}],"predecessor-version":[{"id":15488,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/8023\/revisions\/15488"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=8023"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=8023"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=8023"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=8023"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}