> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://www.comet.com/docs/opik/reference/rest-api/automation-rule-evaluators/create-automation-rule-evaluator/llms.txt.

# Create automation rule evaluator

POST http://localhost:5173/api/v1/private/automations/evaluators
Content-Type: application/json

Create automation rule evaluator

Reference: https://www.comet.com/docs/opik/reference/rest-api/automation-rule-evaluators/create-automation-rule-evaluator

## Servers

- `http://localhost:5173/api` (Local server, default)
- `https://www.comet.com/opik/api` (Opik Cloud)

## Request

### Body (application/json)

This endpoint expects an AutomationRuleEvaluator_Write.

- `AutomationRuleEvaluator_Write`
  - `type`: `llm_as_judge` (llm_as_judge)
    - `action` (enum, required)
      - Allowed values: `evaluator`, `annotation_queue_router`
    - `code` (LlmAsJudgeCode_Write, required)
    - `name` (string, required)
    - `enabled` (boolean, optional)
    - `filters` (list of TraceFilter_Write, optional)
    - `project_id` (string, optional) — Primary project ID (legacy field for backwards compatibility)
    - `project_ids` (list of string, optional) — Project IDs for write operations (used when creating/updating rules)
    - `sampling_rate` (float, optional) — Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.
    - `trigger_scope` (enum, optional) — Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.
      - Allowed values: `production`, `experiment`, `both`
  - `type`: `user_defined_metric_python` (user_defined_metric_python)
    - `action` (enum, required)
      - Allowed values: `evaluator`, `annotation_queue_router`
    - `code` (UserDefinedMetricPythonCode_Write, required)
    - `name` (string, required)
    - `enabled` (boolean, optional)
    - `filters` (list of TraceFilter_Write, optional)
    - `project_id` (string, optional) — Primary project ID (legacy field for backwards compatibility)
    - `project_ids` (list of string, optional) — Project IDs for write operations (used when creating/updating rules)
    - `sampling_rate` (float, optional) — Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.
    - `trigger_scope` (enum, optional) — Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.
      - Allowed values: `production`, `experiment`, `both`
  - `type`: `trace_thread_llm_as_judge` (trace_thread_llm_as_judge)
    - `action` (enum, required)
      - Allowed values: `evaluator`, `annotation_queue_router`
    - `code` (TraceThreadLlmAsJudgeCode_Write, required)
    - `name` (string, required)
    - `enabled` (boolean, optional)
    - `filters` (list of TraceThreadFilter_Write, optional)
    - `project_id` (string, optional) — Primary project ID (legacy field for backwards compatibility)
    - `project_ids` (list of string, optional) — Project IDs for write operations (used when creating/updating rules)
    - `sampling_rate` (float, optional) — Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.
    - `trigger_scope` (enum, optional) — Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.
      - Allowed values: `production`, `experiment`, `both`
  - `type`: `trace_thread_user_defined_metric_python` (trace_thread_user_defined_metric_python)
    - `action` (enum, required)
      - Allowed values: `evaluator`, `annotation_queue_router`
    - `code` (TraceThreadUserDefinedMetricPythonCode_Write, required)
    - `name` (string, required)
    - `enabled` (boolean, optional)
    - `filters` (list of TraceThreadFilter_Write, optional)
    - `project_id` (string, optional) — Primary project ID (legacy field for backwards compatibility)
    - `project_ids` (list of string, optional) — Project IDs for write operations (used when creating/updating rules)
    - `sampling_rate` (float, optional) — Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.
    - `trigger_scope` (enum, optional) — Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.
      - Allowed values: `production`, `experiment`, `both`
  - `type`: `span_llm_as_judge` (span_llm_as_judge)
    - `action` (enum, required)
      - Allowed values: `evaluator`, `annotation_queue_router`
    - `code` (SpanLlmAsJudgeCode_Write, required)
    - `name` (string, required)
    - `enabled` (boolean, optional)
    - `filters` (list of SpanFilter_Write, optional)
    - `project_id` (string, optional) — Primary project ID (legacy field for backwards compatibility)
    - `project_ids` (list of string, optional) — Project IDs for write operations (used when creating/updating rules)
    - `sampling_rate` (float, optional) — Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.
    - `trigger_scope` (enum, optional) — Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.
      - Allowed values: `production`, `experiment`, `both`
  - `type`: `span_user_defined_metric_python` (span_user_defined_metric_python)
    - `action` (enum, required)
      - Allowed values: `evaluator`, `annotation_queue_router`
    - `code` (SpanUserDefinedMetricPythonCode_Write, required)
    - `name` (string, required)
    - `enabled` (boolean, optional)
    - `filters` (list of SpanFilter_Write, optional)
    - `project_id` (string, optional) — Primary project ID (legacy field for backwards compatibility)
    - `project_ids` (list of string, optional) — Project IDs for write operations (used when creating/updating rules)
    - `sampling_rate` (float, optional) — Fraction of production (SDK-logged) items this rule scores, from 0 to 1. Trace rules ignore this value for experiment, playground and optimization traces and score them in full; span and thread rules only ever evaluate SDK-logged data.
    - `trigger_scope` (enum, optional) — Controls whether the rule fires on production traces, experiment traces, or both. Defaults to 'production' if omitted.
      - Allowed values: `production`, `experiment`, `both`

## Response

### 201

Created

## Types

### LlmAsJudgeCode_Write

- `model` (LlmAsJudgeModelParameters_Write, required)
- `messages` (list of LlmAsJudgeMessage_Write, required)
- `variables` (map from string to string, required)
- `schema` (list of LlmAsJudgeOutputSchema_Write, required)
- `max_cost_usd` (double, optional)

### TraceFilter_Write

- `field` (string, optional)
- `operator` (enum, optional)
  - Allowed values: `contains`, `not_contains`, `starts_with`, `ends_with`, `=`, `!=`, `>`, `>=`, `<`, `<=`, `is_empty`, `is_not_empty`, `in`, `not_in`
- `key` (string, optional)
- `value` (string, optional)

### UserDefinedMetricPythonCode_Write

- `metric` (string, required)
- `arguments` (map from string to string, required)

### TraceThreadLlmAsJudgeCode_Write

- `model` (LlmAsJudgeModelParameters_Write, required)
- `messages` (list of LlmAsJudgeMessage_Write, required)
- `schema` (list of LlmAsJudgeOutputSchema_Write, required)
- `max_cost_usd` (double, optional)

### TraceThreadFilter_Write

- `field` (string, optional)
- `operator` (enum, optional)
  - Allowed values: `contains`, `not_contains`, `starts_with`, `ends_with`, `=`, `!=`, `>`, `>=`, `<`, `<=`, `is_empty`, `is_not_empty`, `in`, `not_in`
- `key` (string, optional)
- `value` (string, optional)

### TraceThreadUserDefinedMetricPythonCode_Write

- `metric` (string, required)

### SpanLlmAsJudgeCode_Write

- `model` (LlmAsJudgeModelParameters_Write, required)
- `messages` (list of LlmAsJudgeMessage_Write, required)
- `variables` (map from string to string, required)
- `schema` (list of LlmAsJudgeOutputSchema_Write, required)

### SpanFilter_Write

- `field` (string, optional)
- `operator` (enum, optional)
  - Allowed values: `contains`, `not_contains`, `starts_with`, `ends_with`, `=`, `!=`, `>`, `>=`, `<`, `<=`, `is_empty`, `is_not_empty`, `in`, `not_in`
- `key` (string, optional)
- `value` (string, optional)

### SpanUserDefinedMetricPythonCode_Write

- `metric` (string, required)
- `arguments` (map from string to string, required)

### LlmAsJudgeModelParameters_Write

- `name` (string, required)
- `temperature` (double, optional)
- `seed` (integer, optional)
- `custom_parameters` (JsonNode_Write, optional)

### LlmAsJudgeMessage_Write

- `role` (enum, required)
  - Allowed values: `SYSTEM`, `USER`, `AI`, `TOOL_EXECUTION_RESULT`, `CUSTOM`
- `content` (string, optional)
- `content_array` (list of LlmAsJudgeMessageContent_Write, optional)
- `string_content` (boolean, optional)
- `structured_content` (boolean, optional)

### LlmAsJudgeOutputSchema_Write

- `name` (string, required)
- `type` (enum, required)
  - Allowed values: `BOOLEAN`, `INTEGER`, `DOUBLE`
- `description` (string, required)

### JsonNode_Write

### LlmAsJudgeMessageContent_Write

- `type` (string, required)
- `text` (string, optional)
- `image_url` (ImageUrl_Write, optional)
- `video_url` (VideoUrl_Write, optional)
- `audio_url` (AudioUrl_Write, optional)

### ImageUrl_Write

- `url` (string, required)
- `detail` (string, optional)

### VideoUrl_Write

- `url` (string, required)

### AudioUrl_Write

- `url` (string, required)

## Examples

**Request**

```json
{
  "type": "llm_as_judge",
  "action": "evaluator",
  "name": "string"
}
```

**Response**

```json
{}
```

**SDK Code**

```python
import requests

url = "http://localhost:5173/api/v1/private/automations/evaluators"

payload = {
    "type": "llm_as_judge",
    "action": "evaluator",
    "name": "string"
}
headers = {"Content-Type": "application/json"}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'http://localhost:5173/api/v1/private/automations/evaluators';
const options = {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: '{"type":"llm_as_judge","action":"evaluator","name":"string"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "http://localhost:5173/api/v1/private/automations/evaluators"

	payload := strings.NewReader("{\n  \"type\": \"llm_as_judge\",\n  \"action\": \"evaluator\",\n  \"name\": \"string\"\n}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("http://localhost:5173/api/v1/private/automations/evaluators")

http = Net::HTTP.new(url.host, url.port)

request = Net::HTTP::Post.new(url)
request["Content-Type"] = 'application/json'
request.body = "{\n  \"type\": \"llm_as_judge\",\n  \"action\": \"evaluator\",\n  \"name\": \"string\"\n}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("http://localhost:5173/api/v1/private/automations/evaluators")
  .header("Content-Type", "application/json")
  .body("{\n  \"type\": \"llm_as_judge\",\n  \"action\": \"evaluator\",\n  \"name\": \"string\"\n}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'http://localhost:5173/api/v1/private/automations/evaluators', [
  'body' => '{
  "type": "llm_as_judge",
  "action": "evaluator",
  "name": "string"
}',
  'headers' => [
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("http://localhost:5173/api/v1/private/automations/evaluators");
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{\n  \"type\": \"llm_as_judge\",\n  \"action\": \"evaluator\",\n  \"name\": \"string\"\n}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = ["Content-Type": "application/json"]
let parameters = [
  "type": "llm_as_judge",
  "action": "evaluator",
  "name": "string"
] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "http://localhost:5173/api/v1/private/automations/evaluators")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```

> Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.