Annotation Queues

Involving subject matter experts in AI projects is essential because they provide the domain knowledge and contextual judgment that ensures model outputs are accurate, relevant, and aligned with real-world expectations. Annotation Queues in Opik make it simple for subject matter experts (SMEs) to review and annotate agent outputs. This feature streamlines the human-in-the-loop process by providing easy queue management, simple invitation flows, and a distraction-free annotation experience designed for non-technical users.

Annotation Queues are collections of traces or threads that need human review and feedback. They enable you to organize content for review, share with SMEs easily, collect structured feedback, and track progress across all your evaluation workflows.

Creating and Managing Annotation Queues

Each annotation queue is defined by a collection of traces or threads, evaluation instructions, and feedback definitions:

1. Queue Configuration: Set up the queue with clear instructions and scope
2. Content Selection: Add traces or threads that need human review
3. SME Access: Share queue links with subject matter experts for annotation

Setting Up Your First Queue

Navigate to the Annotation Queues page in your project and click Create Queue.

Configure your queue with:

• Name: Clear identification for your queue
• Scope: Choose between traces or threads
• Instructions: Provide context and guidance for reviewers
• Feedback Definitions: Select the metrics SMEs will use for scoring

Adding Content to Your Queue

You can add items to your queue in several ways:

From Traces/Threads Lists:

• Select one or multiple items
• Click Add to -> Add to annotation queue
• Choose an existing queue or create a new one

From Individual Trace/Thread Details:

• Open the trace or thread detail view
• Click Add to -> Add to annotation queue in the actions panel
• Select your target queue

Sharing with Subject Matter Experts

Once your queue is set up, you can share it with SMEs:

Copy Queue Link:

• Click the Share queue button on your queue to copy the queue link
• Share the link directly with SMEs via email, Slack, or other communication tools

SME Annotation Experience

When SMEs access a queue, they experience a streamlined, distraction-free interface designed for efficient review.

The annotation workflow begins with clear instructions and context, allowing SMEs to understand what they’re evaluating and how to provide meaningful feedback.

The SME interface provides:

1. Clean, focused design: No technical jargon or complex navigation
2. Clear instructions: Queue-specific guidance displayed prominently
3. Structured feedback: Predefined metrics with clear descriptions
4. Progress tracking: Visual indicators of completion status
5. Comment system: Optional text feedback for additional context

Annotation Workflow

1. Access the queue: SME clicks the shared link
2. Review content: Examine the trace or thread output
3. Provide feedback: Score using predefined metrics
4. Add comments: Optional text feedback
5. Submit and continue: Move to the next item

Managing Queues programmatically

You can create and manage annotation queues programmatically using the Python or TypeScript SDK. This is useful for automating the process of adding items to queues based on specific criteria.

Creating an Annotation Queue

1
import opik
2
3
client = opik.Opik()
4
5
# Create a traces annotation queue
6
queue = client.create_traces_annotation_queue(
7
    name="High Priority Traces",
8
    description="Traces that need review",
9
    instructions="Check for accuracy and completeness",
10
    feedback_definition_names=["relevance", "accuracy"]
11
)
12
13
print(f"Created queue: {queue.name} (ID: {queue.id})")

Adding Traces to a Queue

1
import opik
2
3
client = opik.Opik()
4
5
# Get an existing traces queue
6
queue = client.get_traces_annotation_queue("queue-id")
7
8
# Search for traces and add them to the queue
9
traces = client.search_traces(
10
  project_name="my-project",
11
  filter_string='feedback_scores.user_frustration > 0.5'
12
)
13
queue.add_traces(traces)
14
15
# For a single trace, wrap it in a list
16
single_trace = client.get_trace_content("trace-id")
17
queue.add_traces([single_trace])

Working with Thread Queues

1
import opik
2
3
client = opik.Opik()
4
5
# Create a threads annotation queue
6
thread_queue = client.create_threads_annotation_queue(
7
    name="Thread Review Queue",
8
    description="Threads needing review"
9
)
10
11
# Get threads and add them to the queue
12
threads_client = client.get_threads_client()
13
important_threads = threads_client.search_threads(
14
  project_name="my-project",
15
  filter_string='tags contains "important"'
16
)
17
thread_queue.add_threads(important_threads)

Updating and Deleting Queues

1
import opik
2
3
client = opik.Opik()
4
5
# Get an existing traces queue
6
queue = client.get_traces_annotation_queue("queue-id")
7
8
# Update queue properties
9
queue.update(
10
    description="Traces that need a thorough review",
11
    instructions="Check the conversation tone"
12
)
13
14
# Remove traces from the queue
15
short_traces = client.search_traces(
16
  project_name="my-project",
17
  filter_string='duration < 10'
18
)
19
queue.remove_traces(short_traces)
20
21
# Delete the queue
22
queue.delete()

1
import opik
2
3
client = opik.Opik()
4
5
# Get all traces annotation queues for a project
6
traces_queues = client.get_traces_annotation_queues(project_name="my-project")
7
8
for queue in traces_queues:
9
    print(f"Traces Queue: {queue.name}, Items: {queue.items_count}")
10
11
# Get all threads annotation queues for a project
12
threads_queues = client.get_threads_annotation_queues(project_name="my-project")
13
14
for queue in threads_queues:
15
    print(f"Threads Queue: {queue.name}, Items: {queue.items_count}")

Listing Annotation Queues

1
import opik
2
3
client = opik.Opik()
4
5
# Get all traces annotation queues for a project
6
traces_queues = client.get_traces_annotation_queues(project_name="my-project")
7
8
for queue in traces_queues:
9
    print(f"Traces Queue: {queue.name}, Items: {queue.items_count}")
10
11
# Get all threads annotation queues for a project
12
threads_queues = client.get_threads_annotation_queues(project_name="my-project")
13
14
for queue in threads_queues:
15
    print(f"Threads Queue: {queue.name}, Items: {queue.items_count}")

Learn more

You can learn more about Opik’s annotation and evaluation features in:

1. Evaluation overview
2. Feedback definitions