Annotate Traces

Building Feedback Loops with Trace Annotations

This video demonstrates how to add human feedback and annotations to your LLM traces, creating a powerful feedback loop for continuous improvement. You’ll learn to mark traces as high or low quality, add specific feedback scores, and build datasets of annotated examples for future evaluation and fine-tuning. Annotations bridge the gap between raw trace data and actionable insights for model improvement.

Key Highlights

• Easy Feedback Addition: Use the feedback scores section and pen icon to quickly add human annotations directly to traces in the Opik UI
• Custom Feedback Definitions: Create categorical or numerical feedback metrics in Configuration → Feedback Definitions to match your specific evaluation needs
• Flexible Scoring Options: Define custom categories (pass/fail, quality ratings) or sliding scales (0-1) based on metrics like accuracy, relevance, or tone
• Smart Filtering: Filter traces by feedback scores to quickly identify high-performing or problematic outputs for targeted analysis
• Dataset Creation: Convert annotated traces directly into datasets for training data collection and model improvement workflows
• Built-in Metrics Library: Access pre-built metrics like answer relevance, Levenshtein distance, and hallucination detection out of the box
• LLM as a Judge: Use automated LLM-based evaluation where a second LLM evaluates responses from your primary model
• Pattern Identification: Annotations help identify performance patterns, create training data, and establish feedback loops between users and developers