{"id":7393,"date":"2023-09-07T10:10:32","date_gmt":"2023-09-07T18:10:32","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7393"},"modified":"2025-04-24T17:14:22","modified_gmt":"2025-04-24T17:14:22","slug":"nlp-for-text-to-image-generators-prompt-generation-part-2","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/nlp-for-text-to-image-generators-prompt-generation-part-2\/","title":{"rendered":"NLP for Text-to-Image Generators \u2014 Prompt Generation [Part 2]"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Src: https:\/\/analyticsindiamag.com\/the-ai-art-generation-tools-that-you-can-actually-use\/<\/a><\/figcaption><\/figure>\n

In the previous tutorial<\/a>, we analyzed a large dataset of prompts created by Midjourney users. By using natural language processing (NLP), we were able to create a semantic search and model topics for the existing topics. In this article, we are going to dive deeper and explore ways of using language models to meaningful generate prompts.<\/p>\n

In order to have the best results, the quality of the prompt describing the visual matters. Prompt engineering is the concept of providing an input to a large language model and it generates a complete prompt or phrase as the output. The generated output depends on the task that the underlying language model was trained on.<\/p>\n

Language models like GPT-3 by OpenAI have been trained using a large number of parameters and can provide good results for general tasks like writing topics for articles. However, these large language models can be fine-tuned by training them on specific data so that they can perform better on specific tasks. For our use case, we\u2019ll need to fine-tune a large language generation model provided by Cohere. This makes it able to generate prompts that can be fed into the text-to-image generators.<\/p>\n

Prerequisites<\/strong><\/h1>\n

You need to have Python 3.6+ installed in your development machine in order to follow along. In addition to this, you need an account to use Cohere\u2019s platform.<\/p>\n

Cohere is a platform that provides access to advanced large language models and NLP tools through one easy-to-use API. The platform offers free credits that you can use to experiment with your NLP projects. Make sure you install the following dependencies.<\/p><\/blockquote>\n

pip install datasets cohere numpy pandas nltk<\/span><\/pre>\n
Data preparation<\/strong><\/h1>\n
You will use the Midjourney\u2019s prompt dataset<\/a> available on Hugging Face. For this tutorial, you need to clean the prompts, add a separator and export them in txt <\/em>format. This is because you will use this data to fine-tune a language model using Cohere\u2019s platform. To get started, first load the dataset.<\/p>\n
from datasets import load_dataset\nimport pandas as pd\ndataset = load_dataset(\u201csuccinctly\/midjourney-prompts\u201d)\ndf = dataset[\u2018train\u2019].to_pandas()<\/span><\/pre>\n
Next, clean the dataset by removing duplicates, prompts with less than 10 words, and empty rows. This ensures the training data is of high quality. The resulting dataset has 61k prompts.<\/p>\n
df = df[df[\u2018text\u2019].str.strip().astype(bool)]\ndf = df.drop_duplicates(subset=[\u2018text\u2019], keep=\u2019first\u2019)<\/span><\/pre>\n
Finally, make sure you write the prompts to a txt <\/em>file. Make sure you add a separator such as \u201c \u2014 END PROMPT \u2014 \u201d to identify different prompts.<\/p>\n
import nltk\nnltk.download('punkt')\n\nwith open('\/path\/to\/folder\/midjourney_prompts.txt', 'w') as f:\n  for index, row in df.iterrows():\n    if len(word_tokenize(row['text'])) > 10:\n      f.write(row['text'])\n      f.write('\\n-- END PROMPT --\\n')<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n
Prompt engineering plus Comet plus Gradio? What comes out is amazing AI-generated art! Take a closer look at our public logging project<\/a> to see some of the amazing creations that have come out of this fun experiment.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n
Fine-tuning Cohere\u2019s generation model<\/strong><\/h1>\n
Cohere has a generation model that you\u2019ll use as the base model to fine-tune it with your training data. The process is simple and user-friendly. Log in to your Cohere account, head over to the dashboard, and click on \u2018Create Finetune.\u2019<\/p>\n
Select \u2018Generation (Generate)\u2019 as your baseline model and \u2018small\u2019 as the model size. The smaller the model, the faster it is, but it has lower performance and vice-versa. Next, upload your txt file that has your training data and configure the data separator as shown below.<\/p>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
Click \u2018Preview Data\u2019 to confirm the samples are correct. If you are satisfied, click \u2018Review Data.\u2019<\/p>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
Give your fine-tune model a name then click \u2018Start Fine-tuning\u2019 to kickstart the process.<\/p>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
Monitor the status of the training process and once it changes to \u2018Ready,\u2019 you can now start using the model.<\/p>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
Prompt generation<\/strong><\/h1>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Src: https:\/\/docs.cohere.ai\/prompt-engineering-wiki\/<\/a><\/figcaption>\n<\/figure>\n
In this section, you will integrate your fine-tuned language model with your prompt generation process. This is where prompt engineering comes to play. By automating this process, you can generate up to five prompts with a single API call and iterate based on results. Also, you can add various parameters to this API call to configure the generation process so that it suits your needs. Visit Cohere\u2019s Generate Documentation<\/a> for more information about these configuration options.<\/p>\n
First, you\u2019ll need to grab your API key from Cohere\u2019s dashboard and initialize the client. Kindly note that you need to specify the version so that some features are available.<\/p>\n
import cohere\nco = cohere.Client(\u2018{apiKey}\u2019, \u20182021\u201311\u201308\u2019)<\/span><\/pre>\n
Next, call the generate API. In this call, you will specify the following parameters:<\/p>\n