{"id":11633,"date":"2024-10-08T11:33:53","date_gmt":"2024-10-08T19:33:53","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=11633"},"modified":"2026-01-09T18:27:56","modified_gmt":"2026-01-09T18:27:56","slug":"openai-evals","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/openai-evals\/","title":{"rendered":"OpenAI Evals: Log Datasets & Evaluate LLM Performance with Opik"},"content":{"rendered":"\n

 <\/p>\n\n\n\n

\"featured<\/figure>\n\n\n\n

OpenAI\u2019s Python API is quickly becoming one of the most-downloaded Python packages. With an easy-to-use SDK and access to a reliable hardware infrastructure, OpenAI\u2019s Python API is widely considered one of the best tools for developers working on LLM-powered applications. Whether you are building a chatbot, a summarization tool, a machine translation system or a sentiment classifier, OpenAI\u2019s Python API makes it easy to start prototyping.<\/p>\n\n\n\n

There is a famous saying by Lord Kelvin: \u201cIf you can\u2019t measure it, you can\u2019t improve it.\u201d Building a robust LLM-powered application requires a combination of rigorous experimentation and constant LLM evaluation<\/a>. Developers are continuously iterating on their prompts so that their applications achieve more desirable outputs from the LLMs they are calling. Simultaneously, developers need a way to quantify the quality of their LLM responses via scoring and annotation processes in order to see what is and isn\u2019t working.<\/p>\n\n\n\n

Some of these processes require manual effort, like human-in-the-loop<\/a> to check a set of LLM responses and leave notes or scores that describe their accuracy or desirability. To improve efficiency at scale, automated scoring methods can be combined with human feedback. Simple deterministic functions can evaluate for specific requirements. (For example, in a use case that requires responses in JSON from the LLM, a function could evaluate whether each response contains valid JSON, leaving a score of 1 for yes and 0 for no.) A secondary LLM (LLM-as-a-judge<\/a>) can also be used to score more complex qualities like factuality, answer relevance, and context precision.<\/p>\n\n\n\n

Opik, by Comet, is an LLM evaluation framework<\/a> that developers rely on in both development and production to measure the performance of their LLMs using the techniques described above, and more. With the ability to track LLM inputs, outputs and any related metadata, Opik serves as a single system of record for your prompt engineering<\/a> work. Opik also allows you to run eval experiments so you can quantitatively compare your LLM responses to see which prompt templates, models, and hyper-parameters produce the best results.<\/p>\n\n\n\n

The best part? Opik has a native integration with OpenAI, meaning with just a couple of lines of code you can get out-of-box logging of all your OpenAI interactions to the Opik platform.<\/p>\n\n\n\n

 <\/p>\n\n\n\n

Start Logging OpenAI Datasets<\/h2>\n\n\n\n

Opik is completely open-source. You can choose to self-host it by following the instructions listed here, or you can sign up for an account on our hosted version.<\/p>\n\n\n\n

To install the Opik SDK in your python virtual environment, run the following command:<\/p>\n\n\n\n

pip install opik\n<\/code><\/pre>\n\n\n\n
Below is an example of how easy it is to set up logging with Opik.<\/p>\n\n\n\n