{"id":7426,"date":"2023-09-13T14:56:08","date_gmt":"2023-09-13T22:56:08","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7426"},"modified":"2025-04-24T17:14:03","modified_gmt":"2025-04-24T17:14:03","slug":"organize-your-prompt-engineering-with-cometllm","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/","title":{"rendered":"Organize Your Prompt Engineering with CometLLM"},"content":{"rendered":"\n

Introduction<\/h2>\n\n\n\n

Prompt Engineering<\/a> is arguably the most critical aspect in harnessing the power of Large Language Models (LLMs) like ChatGPT. Whether you\u2019re building tools for content creation, question answering, translation, or coding assistance, well designed prompts act as the roadmap for LLMs to generate accurate, ethical, and contextually relevant outputs. However; current prompt engineering workflows are incredibly tedious and cumbersome. Logging prompts and their outputs to .csv files and then visualizing them in Excel becomes inefficient and confusing. Developers are often experimenting with hundreds if not thousands of prompts for a given use-case and are in dire need of tools that scale well!<\/p>\n\n\n\n

Comet is happy to announce a new solution that aims to make the entire prompt engineering process much easier. Use CometLLM to log and visualize all your prompts and chains and unleash the full potential of Large Language Models!<\/p>\n\n\n\n

P.S: If you prefer to learn by code, check out our informative Colab notebook<\/a> where we show you how to log prompts from open-source HuggingFace Models!<\/p>\n\n\n\n

\"Comet_LLM<\/figure>\n\n\n\n

Get Started With CometLLM<\/h2>\n\n\n\n

Integrating CometLLM into your prompt experimentation workflows is seamless!<\/p>\n\n\n\n

First install the package via pip.<\/p>\n\n\n\n

pip install comet_llm<\/span><\/pre>\n\n\n\n
Then, head over to Comet to create a free account<\/a> and get your API key! Once you have that, you\u2019re all set to start logging prompts and their outputs to Comet.<\/p>\n\n\n\n
import<\/span> comet_llm\n\ncomet_llm.log_prompt(\n    prompt=\"What is your name?\"<\/span>,\n    output=\" My name is Alex.\"<\/span>,\n    api_key=\"YOUR_COMET_API_KEY\"<\/span>,\n    project = \"YOUR_LLM_PROJECT\"<\/span>,\n)<\/span><\/pre>\n\n\n\n
\n
\n
\n
Add Token Usage to Prompt Metadata<\/h2>\n
Prompt usage tokens refer to the number of tokens within a language model\u2019s input that are consumed by the prompts or instructions provided to the model. In the context of OpenAI\u2019s GPT model, the total number of tokens in an input affects the cost, response time, and availability of the model. Log your Usage Token as metadata to your prompts to make sure you are using the most cost-effective prompts for your use-case!<\/p>\n
\n
\n
\"Examining<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
Score Prompt Outputs<\/strong><\/h2>\n
Prompt Feedback is crucial for improving the overall quality of Large Language Models. Use the Comet UI to give human-feedback on a prompt and it will automatically document your score!<\/p>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
Evaluate Prompt Templates<\/h2>\n
Prompt Templates are predefined structures or formats that guide users in providing input to a language model. Templates help in improving the clarity, consistency, and the overall quality of LLM responses. Users can log the prompt template, along with their prompt input and outputs to decide which templates are providing better responses.<\/p>\n
import<\/span> comet_llm\n\ncomet_llm.log_prompt(\n    prompt=\"Answer the question and if the question can't be answered, say \\\"I don't know\\\"\\n\\n---\\n\\nQuestion: What is your name?\\nAnswer:\"<\/span>,\n    prompt_template=\"Answer the question and if the question can't be answered, say \\\"I don't know\\\"\\n\\n---\\n\\nQuestion: {{question}}?\\nAnswer:\"<\/span>,\n    prompt_template_variables={\"question\"<\/span>: \"What is your name?\"<\/span>},\n    metadata= {\n        \"usage.prompt_tokens\"<\/span>: 7<\/span>,\n        \"usage.completion_tokens\"<\/span>: 5<\/span>,\n        \"usage.total_tokens\"<\/span>: 12<\/span>,\n    },\n    output=\" My name is Alex.\"<\/span>,\n    duration=16.598<\/span>,\n)<\/span><\/pre>\n
Search for Specific Prompts<\/h2>\n
Prompt Engineering is an iterative process. Developers are often trying hundreds if not thousands of prompts. CometLLM makes it easy to find specific prompts with our search functionality.<\/a> Search for a keyword within a prompt input, output or template<\/strong> and quickly find the relevant prompts!<\/p>\n
\n
\n
\"Organizing<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
Compare and Contrast LLMs<\/h2>\n
There are many powerful LLMs out there! (GPT-3.5 , LLama 2, Falcon, Davinci). But which one is the best for a particular use-case? Tag your prompt responses with the LLM that generated them. Then use CometLLM\u2019s group-by functionality to easily group prompts by model and decide which one is generating the highest quality responses!<\/p>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
Visualize Chat History With Chains<\/h2>\n
Prompt chains refer to a sequence of prompts that are used to guide a language model\u2019s response in a conversational or interactive setting. Instead of providing a single prompt, prompt chains involve sending multiple prompts in succession to maintain context and guide the model\u2019s behavior throughout a conversation. Log your interactions with chatbots like ChatGPT as chains <\/a>to quickly track and debug a LLM\u2019s train of thought.<\/p>\n
\n
\n
\"Comet_LLM<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
Check-out the Github Repo<\/h2>\n
CometLLM\u2019s SDK is completely open-sourced and can be found on Github<\/a>! Give the project a star and make sure to keep following the repository for exciting product updates!<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Introduction Prompt Engineering is arguably the most critical aspect in harnessing the power of Large Language Models (LLMs) like ChatGPT. Whether you\u2019re building tools for content creation, question answering, translation, or coding assistance, well designed prompts act as the roadmap for LLMs to generate accurate, ethical, and contextually relevant outputs. However; current prompt engineering workflows are […]<\/p>\n","protected":false},"author":21,"featured_media":7513,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[8,65,9],"tags":[40,14,64,30,15,52,31,32,56,34],"coauthors":[134],"class_list":["post-7426","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-comet-community-hub","category-llmops","category-product","tag-comet","tag-comet-ml","tag-cometllm","tag-deep-learning","tag-deep-learning-experiment-management","tag-llm","tag-llmops","tag-nlp","tag-optimization","tag-prompt-engineering"],"yoast_head":"\nOrganize Your Prompt Engineering with CometLLM - Comet<\/title>\n<meta name=\"description\" content=\"Streamline your Prompt Engineering Workflows with CometLLM! Log, Debug, and Rate your LLM Prompts and Chains.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Organize Your Prompt Engineering with CometLLM\" \/>\n<meta property=\"og:description\" content=\"Streamline your Prompt Engineering Workflows with CometLLM! Log, Debug, and Rate your LLM Prompts and Chains.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-09-13T22:56:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:14:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/comet_llm_screenshot-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1346\" \/>\n\t<meta property=\"og:image:height\" content=\"850\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Siddharth Mehta\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Siddharth Mehta\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Organize Your Prompt Engineering with CometLLM - Comet","description":"Streamline your Prompt Engineering Workflows with CometLLM! Log, Debug, and Rate your LLM Prompts and Chains.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/","og_locale":"en_US","og_type":"article","og_title":"Organize Your Prompt Engineering with CometLLM","og_description":"Streamline your Prompt Engineering Workflows with CometLLM! Log, Debug, and Rate your LLM Prompts and Chains.","og_url":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-09-13T22:56:08+00:00","article_modified_time":"2025-04-24T17:14:03+00:00","og_image":[{"width":1346,"height":850,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/comet_llm_screenshot-1.png","type":"image\/png"}],"author":"Siddharth Mehta","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Siddharth Mehta","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/"},"author":{"name":"Siddharth Mehta","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/652eb7d782d18f295922f50ea3b9e54c"},"headline":"Organize Your Prompt Engineering with CometLLM","datePublished":"2023-09-13T22:56:08+00:00","dateModified":"2025-04-24T17:14:03+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/"},"wordCount":622,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/comet_llm_screenshot-1.png","keywords":["Comet","Comet ML","CometLLM","Deep Learning","Deep Learning Experiment Management","LLM","LLMOps","NLP","Optimization","Prompt Engineering"],"articleSection":["Comet Community Hub","LLMOps","Product"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/","url":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/","name":"Organize Your Prompt Engineering with CometLLM - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/comet_llm_screenshot-1.png","datePublished":"2023-09-13T22:56:08+00:00","dateModified":"2025-04-24T17:14:03+00:00","description":"Streamline your Prompt Engineering Workflows with CometLLM! Log, Debug, and Rate your LLM Prompts and Chains.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/comet_llm_screenshot-1.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/comet_llm_screenshot-1.png","width":1346,"height":850,"caption":"CometLLM platform screenshot"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/organize-your-prompt-engineering-with-cometllm\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Organize Your Prompt Engineering with CometLLM"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/652eb7d782d18f295922f50ea3b9e54c","name":"Siddharth Mehta","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/940c7280faea9e1b8b086c2ed7ec01db","url":"https:\/\/secure.gravatar.com\/avatar\/27a672e997fa7a66796e4be0503e0efeec6bd34daae185bb6de163227a5a0739?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/27a672e997fa7a66796e4be0503e0efeec6bd34daae185bb6de163227a5a0739?s=96&d=mm&r=g","caption":"Siddharth Mehta"},"description":"ML Growth Engineer @ Comet. Interested in Computer Vision, Robotics, and Reinforcement Learning","sameAs":["https:\/\/www.comet.com\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/siddharthmcomet-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7426","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7426"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7426\/revisions"}],"predecessor-version":[{"id":15540,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7426\/revisions\/15540"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/7513"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7426"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7426"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7426"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7426"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}