Message Processing Emulation Models¶

This module provides data models used for message processing emulation in Opik. These models represent the core data structures for traces, spans, and feedback scores that are used internally by the Opik SDK during evaluation.

Overview¶

The message processing emulation models are primarily used in evaluation contexts, particularly for task span evaluation where custom metrics need access to detailed execution information. These models provide a structured representation of:

Traces: Complete execution paths of requests or operations
Spans: Individual steps or operations within a trace
Feedback Scores: Evaluation results attached to traces and spans
Experiment Items: Links between traces, datasets, and experiment runs

Key Classes¶

Class Hierarchy¶

The models form a hierarchical relationship:

TraceModel
├── spans: List[SpanModel]
│   ├── spans: List[SpanModel]  (nested spans)
│   └── feedback_scores: List[FeedbackScoreModel]
└── feedback_scores: List[FeedbackScoreModel]

Quick Start¶

Import the models:

from opik.message_processing.emulation.models import (
    TraceModel,
    SpanModel,
    FeedbackScoreModel,
    ExperimentItemModel
)

Common Usage Patterns¶

Task Span Evaluation¶

The primary use case for these models is in task span evaluation, where custom metrics analyze span data:

from opik.evaluation.metrics import BaseMetric, score_result
from opik.message_processing.emulation.models import SpanModel

class CustomSpanMetric(BaseMetric):
    def score(self, task_span: SpanModel) -> score_result.ScoreResult:
        # Access span properties
        span_name = task_span.name
        input_data = task_span.input
        output_data = task_span.output

        # Perform evaluation logic
        score_value = self.evaluate_span(span_name, input_data, output_data)

        return score_result.ScoreResult(
            value=score_value,
            name=self.name,
            reason=f"Evaluated span: {span_name}"
        )

Analyzing Trace Structure¶

You can traverse and analyze the hierarchical structure of traces:

def analyze_trace_structure(trace: TraceModel):
    print(f"Trace: {trace.name}")
    print(f"Total spans: {len(trace.spans)}")

    for span in trace.spans:
        print(f"  Span: {span.name} (type: {span.type})")

        # Analyze nested spans
        for nested_span in span.spans:
            print(f"    Nested: {nested_span.name}")

Working with Feedback Scores¶

Both traces and spans can contain feedback scores from evaluations:

def collect_all_scores(trace: TraceModel):
    all_scores = []

    # Collect trace-level scores
    all_scores.extend(trace.feedback_scores)

    # Collect span-level scores
    for span in trace.spans:
        all_scores.extend(span.feedback_scores)

        # Recursively collect from nested spans
        for nested_span in span.spans:
            all_scores.extend(nested_span.feedback_scores)

    return all_scores

Integration with Evaluation System¶

These models are automatically populated and used by the Opik evaluation system:

Trace Creation: When you run opik.evaluate(), traces are automatically created
Span Population: Individual function calls become spans within the trace
Task Span Evaluation: Metrics with task_span parameters receive SpanModel objects
Score Attachment: Feedback scores are automatically attached to the appropriate traces and spans

You typically don’t need to create these models manually - they’re generated automatically during evaluation. However, understanding their structure is essential for writing effective task span evaluation metrics.

Use Cases¶

These models are commonly used for:

Custom Evaluation Metrics: Analyzing detailed execution data in custom metrics
Performance Analysis: Understanding execution patterns and performance characteristics
Debugging: Investigating issues in complex operations
Cost Tracking: Aggregating usage and cost information across operations
Quality Assessment: Evaluating the quality of individual steps and overall operations

Module Reference¶

For detailed API documentation, see the following class reference pages: