PCA<\/a>). Create two functions \u2014 one for creating the principal component analysis, and the other one for plotting a scatter plot:<\/p>\nfrom sklearn.decomposition import PCA\nimport altair as alt\n\n# Compute the principal components\ndef get_pc(arr,n):\n pca = PCA(n_components=n)\n embeds_transform = pca.fit_transform(arr)\n return embeds_transform\n\n# Generate scatter plots\ndef scatter_plot(df,xcol,ycol,color='basic',title=''):\n chart = alt.Chart(df).mark_circle(size=500).encode(\n x= alt.X(xcol,\n scale=alt.Scale(zero=False),\n axis=alt.Axis(labels=False, ticks=False, domain=False)\n ),\n y= alt.Y(ycol,\n scale=alt.Scale(zero=False),\n axis=alt.Axis(labels=False, ticks=False, domain=False)\n ),\n color= alt.value('#333293') if color == 'basic' else color,\n tooltip=['text']\n )\n result = chart.configure(background=\"#FFF\"\n ).properties(\n width=800,\n height=500,\n title=title\n ).configure_legend(\n orient='bottom', titleFontSize=18,labelFontSize=18)\n return result<\/pre>\nWith the two functions created, calculate the principal components using the word embeddings, then plot them:<\/p>\n
embeds = np.array(df[\u2018text_embeds\u2019].tolist())\nembeds_pc = get_pc(embeds,2)<\/span># Append the principal components to dataframe\ndf = pd.concat([df, pd.DataFrame(embeds_pc)], axis=1)<\/span># Plot the 2D embeddings on a chart\ndf.columns = df.columns.astype(str)\nsample = 500<\/span><\/pre>\nThe scatter plot generated displays all the text prompts analyzed. Prompts that have similar meanings are closer to each other.<\/p>\n\n\n
<\/picture><\/div>\n<\/div>\n<\/figure>\nSemantic search for prompts<\/strong><\/h1>\nThis is a technique that will allow you to find prompts that have a similar meaning to your search query. It goes beyond returning results that match keywords in the search query, by utilizing the word embeddings created above and calculating similarity based on physical distance. First, create a function that calculates a similarity score between two given word embeddings:<\/p>\n
from sklearn.metrics.pairwise import cosine_similarity\n\ndef get_similarity(target,candidates):\n # Turn list into array\n candidates = np.array(candidates)\n target = np.expand_dims(np.array(target),axis=0)\n\n # Calculate cosine similarity\n sim = cosine_similarity(target,candidates)\n sim = np.squeeze(sim).tolist()\n sort_index = np.argsort(sim)[::-1]\n sort_score = [sim[i] for i in sort_index]\n similarity_scores = zip(sort_index,sort_score)\n\n # Return similarity scores\n return similarity_scores<\/pre>\nNext, create word embeddings for your search query and compute the similarity scores between them and the embeddings of our prompts:<\/p>\n
query = \"cartoon animals high definition\"\n# embeddings of the search query\nquery_embeds = co.embed( model='small', texts=[query]).embeddings[0]\n# similarity between the query and existing prompts\nsimilarity = get_similarity(search_query_embeds,embeds[:sample])\n\nprint('Similar prompts:')\nfor idx,sim in similarity:\n if sim >= 0.30:\n df.at[idx,'similar'] = 'yes'\n else:\n df.at[idx,'similar'] = 'no'\n print(f'Similarity: {sim:.2f};',df.iloc[idx]['text'])<\/pre>\nYou can display a scatter plot to visualize the similarity of prompts. You\u2019ll notice that similar results generally appear closer to each other.<\/p>\n\n\n
<\/picture><\/div>\n<\/div>\n<\/figure>\nTopic clustering<\/strong><\/h1>\nIn this final section, we will explore the various topics in the prompts dataset. We will use the KMeans algorithm to create ten clusters of prompts. This algorithm is unsupervised, meaning the clusters are not labeled. However, you can plot word clouds to identify top keywords in each cluster and manually label these afterwards if you choose.<\/p>\n
First, use scikit-learn library to set the number of clusters and fit the model:<\/p>\n
from sklearn.cluster import KMeans<\/span>n_clusters=5<\/span># Cluster the embeddings\nkmeans_model = KMeans(n_clusters=n_clusters, random_state=0)\nclasses = kmeans_model.fit_predict(embeds).tolist()\ndf[\u2018cluster\u2019] = (list(map(str,classes)))<\/span><\/pre>\nNext, create word clouds for each cluster and plot them. Observe and note the main topic for each cluster:<\/p>\n
from wordcloud import WordCloud, STOPWORDS\nimport matplotlib.pyplot as plt\nstopwords = set(STOPWORDS)\n\nfor n in range(n_clusters):\n df_wordcloud = df.loc[df['cluster'] == str(n)]\n text = \" \".join(i for i in df_wordcloud.text)\n wordcloud = WordCloud(width = 800, height = 800,\n background_color ='white',\n stopwords = stopwords,\n min_font_size = 10).generate(text)\n plt.figure(figsize = (8, 8), facecolor = None)\n plt.imshow(wordcloud)\n plt.axis(\"off\")\n plt.tight_layout(pad = 0)\n\n # plt.show()<\/pre>\n\n<\/picture><\/div>\n<\/figure>\nFinally, label the clusters and generate the scatter plot.<\/p>\n
# labels the clusters after looking at keywords in each clusterdf['cluster'] = df['cluster'].replace([\"0\",'1','2','3','4'],\n ['optimization','datasets',\n 'reasoning', 'manipulation',\n 'NLP'])\ndf.columns = df.columns.astype(str)\nscatter_plot(df.iloc[:sample],'0','1',color='cluster',\n title='Prompts clustered into 5 topics')<\/span><\/pre>\n\n\n
<\/picture><\/div>\n<\/div>\n<\/figure>\nConclusion<\/strong><\/h1>\nText-to-image generators present many opportunities for people in the creative fields. We have learned some of the top AI solutions currently in the market and how they work. Additionally, by analyzing a large dataset of prompts, we were able to understand how to create our own prompts and calculate the similarity of prompts. Through semantic search, we can find prompt ideas that are close to what we are looking to achieve. We can extend this analysis by adding sentiment analysis and add mood to the mix. Don\u2019t miss the next part of this tutorial, where we are going to train a prompt generator that creates prompts for us!<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Art by DALL-E 2 Text-to-image generators are currently a red hot topic in the field of AI art. With them, a user can provide text describing the artwork they\u2019d like to output, and the machine generates different variations of this image (sometimes in less than a minute!). Deep learning powers this technology, which helps anyone […]<\/p>\n","protected":false},"author":65,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[165],"class_list":["post-7054","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\n
NLP for Text-to-Image Generators: Prompt Analysis [Part 1] - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"NLP for Text-to-Image Generators: Prompt Analysis [Part 1]\" \/>\n<meta property=\"og:description\" content=\"Art by DALL-E 2 Text-to-image generators are currently a red hot topic in the field of AI art. With them, a user can provide text describing the artwork they\u2019d like to output, and the machine generates different variations of this image (sometimes in less than a minute!). Deep learning powers this technology, which helps anyone […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-08-08T19:46:11+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:14:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uY2VNRs34bfnz0DfSOeSfw.jpeg\" \/>\n<meta name=\"author\" content=\"Klurdy Studios\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Klurdy Studios\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"NLP for Text-to-Image Generators: Prompt Analysis [Part 1] - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/","og_locale":"en_US","og_type":"article","og_title":"NLP for Text-to-Image Generators: Prompt Analysis [Part 1]","og_description":"Art by DALL-E 2 Text-to-image generators are currently a red hot topic in the field of AI art. With them, a user can provide text describing the artwork they\u2019d like to output, and the machine generates different variations of this image (sometimes in less than a minute!). Deep learning powers this technology, which helps anyone […]","og_url":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-08-08T19:46:11+00:00","article_modified_time":"2025-04-24T17:14:57+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uY2VNRs34bfnz0DfSOeSfw.jpeg","type":"","width":"","height":""}],"author":"Klurdy Studios","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Klurdy Studios","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/"},"author":{"name":"Klurdy Studios","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/069e186ad4a5b6d6950292821ea0f37b"},"headline":"NLP for Text-to-Image Generators: Prompt Analysis [Part 1]","datePublished":"2023-08-08T19:46:11+00:00","dateModified":"2025-04-24T17:14:57+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/"},"wordCount":968,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uY2VNRs34bfnz0DfSOeSfw.jpeg","articleSection":["Machine Learning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/","url":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/","name":"NLP for Text-to-Image Generators: Prompt Analysis [Part 1] - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uY2VNRs34bfnz0DfSOeSfw.jpeg","datePublished":"2023-08-08T19:46:11+00:00","dateModified":"2025-04-24T17:14:57+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uY2VNRs34bfnz0DfSOeSfw.jpeg","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uY2VNRs34bfnz0DfSOeSfw.jpeg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-analysis-part-1\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"NLP for Text-to-Image Generators: Prompt Analysis [Part 1]"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/069e186ad4a5b6d6950292821ea0f37b","name":"Klurdy Studios","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/b1a3bf3caaa793aaad2da005b3ba38ba","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/1635710213869-96x96.jpg","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/1635710213869-96x96.jpg","caption":"Klurdy Studios"},"url":"https:\/\/www.comet.com\/site\/blog\/author\/brianklurdy-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7054","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/65"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7054"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7054\/revisions"}],"predecessor-version":[{"id":15589,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7054\/revisions\/15589"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7054"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7054"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7054"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7054"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uY2VNRs34bfnz0DfSOeSfw.jpeg\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*mxb3sNfokvSQi_P0qSoSEA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*9GN8fG0LTDqdhbTCNuUJZQ.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*oVN4zWGNUOCdGpuawy9x-g.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:600\/1*y1dX9b3FJXUUTeq7D1uRJA.gif\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Zrs9Tz0wt5e9Ax2UUl4Tjg.png\")
<\/figure>