Track Large Language Models¶

When it comes to building applications using Large Language Models, a lot of time is spent working on prompt engineering rather than training models. This new workflow requires a different set of tools that Comet is developing under the umbrella of LLMOps.

The tools available for Large Language Models fall under two categories:

• LLM projects: Dedicated views specifically for analyzing prompts, responses and chaings. The functionality has been specifically built to allow you to analyze tens of thousands of prompts.
• LLM panels: Visualizations that can be used with Experiment Management to view prompts and chains, usefull when projects contain both fine-tuning and prompt engineering use-cases.

LLM Projects¶

LLM Projects differ from standard Experiment Management projects in that they are customized to the prompt engineering workflow rather than a fine-tuning or model training workflow.

Creating an LLM project can be done in one of two ways:

• Through the Comet UI by using the New Project button and choosing Large Language Model as the project type.
• By using the LLM Python SDK to log a prompt to a new project.

Once the LLM project has been created, it can be accessed from the Projects page and gives you access to a number of prompt analysis tools.

Learn more about LLM projects here.

Example projects¶

A demo LLM project is available here: LLM demo project