AI Providers

AI Providers let you connect LLMs for use in the Playground and Online Evaluation.

Using AI Providers

Once configured, providers appear in two places:

1. Playground — Test prompts interactively with different models.
2. Online Evaluation — Run LLM-as-a-judge scoring on your traces.

Adding a Provider

1. Click the Add configuration button in the top-right corner

2. In the Provider Configuration dialog that appears:
   • Select a provider
   • Enter your API key for that provider
   • Click Save to store the configuration

Supported Providers

Opik supports integration with various AI providers, including:

• OpenAI
• Anthropic
• OpenRouter
• Gemini
• VertexAI
• Azure OpenAI
• Amazon Bedrock
• Ollama (local or self-hosted, OpenAI-compatible)
• vLLM / any other OpenAI API-compliant provider

Provider-Specific Setup

Below are instructions for obtaining API keys and other required information for each supported provider:

OpenAI

1. Create or log in to your OpenAI account
2. Navigate to the API keys page
3. Click “Create new secret key”
4. Copy your API key (it will only be shown once)
5. In Opik, select “OpenAI” as the provider and paste your key
6. (Optional) Under Custom headers, add any static HTTP headers to send with every request to OpenAI (see the note under OpenRouter for the headers you can’t set)

Anthropic

1. Sign up for or log in to Anthropic’s platform
2. Navigate to the API Keys page
3. Click “Create Key” and select the appropriate access level
4. Copy your API key (it will only be shown once)
5. In Opik, select “Anthropic” as the provider and paste your key

OpenRouter

1. Create or log in to your OpenRouter account
2. Navigate to the API Keys page
3. Create a new API key
4. Copy your API key
5. In Opik, select “OpenRouter” as the provider and paste your key
6. (Optional) Under Custom headers, add any static HTTP headers to send with every request. For example, OpenRouter’s app attribution headers make Opik calls show up under your app name in OpenRouter’s activity view:
   • HTTP-Referer: https://your-app.example.com
   • X-OpenRouter-Title: Your App Name

Gemini

1. Signup or login to Google AI Studio
2. Go to the API keys page\
3. Create a new API key for one your existing Google Cloud project
4. Copy your API key (it will only be shown once)
5. In Opik, select “Gemini” as the provider and paste your key

Azure OpenAI

Azure OpenAI provides enterprise-grade access to OpenAI models through Microsoft Azure. To use Azure OpenAI with Opik:

1. Get your Azure OpenAI endpoint URL

   • Go to portal.azure.com
   • Navigate to your Azure OpenAI resource
   • Copy your endpoint URL (it looks like https://your-company.openai.azure.com)

2. Construct the complete API URL

   • Add /openai/v1 to the end of your endpoint URL
   • Your complete URL should look like: https://your-company.openai.azure.com/openai/v1

3. Configure in Opik

   • In Opik, go to Workspace Settings > AI Providers
   • Click “Add Configuration”
   • Select “vLLM / Custom provider” from the dropdown
   • Enter your complete URL in the URL field: https://your-company.openai.azure.com/openai/v1
   • Add your Azure OpenAI API key in the API Key field
   • In the Models section, list the models you have deployed in Azure (e.g., gpt-4o)
   • Click Save to store the configuration

Once saved, you can use your Azure OpenAI models directly from Online Scores and the Playground.

Vertex AI

Option A: Setup via gcloud CLI

1. Create a Custom IAM Role

gcloud iam roles create opik \
  --project=<my-project> \
  --title="Opik" \
  --description="Custom IAM role for Opik" \
  --permissions=aiplatform.endpoints.predict,resourcemanager.projects.get \
  --stage=ALPHA

2. Create a Service Account

gcloud iam service-accounts create opik-sa \
  --description="Service account for Opik role" \
  --display-name="Opik Service Account"

3. Assign the Role to the Service Account

gcloud projects add-iam-policy-binding <my-project> \
  --member="serviceAccount:opik-sa@<my-project>.iam.gserviceaccount.com" \
  --role="projects/<my-project>/roles/opik"

4. Generate the Service Account Key File

gcloud iam service-accounts keys create opik-key.json \
  --iam-account=opik-sa@<my-project>.iam.gserviceaccount.com

The file opik-key.json contains your credentials. Open it in a text editor and copy the entire contents.

Option B: Setup via Google Cloud Console (UI)

Step 1: Create the Custom Role

1. Go to IAM > Roles
2. Click Create Role
3. Fill in the form:

• Title: Opik
• ID: opik
• Description: Custom IAM role for Opik
• Stage: Alpha

4. Click Add Permissions, then search for and add:

• aiplatform.endpoints.predict
• resourcemanager.projects.get

5. Click Create

Step 2: Create the Service Account

1. Go to IAM > Service Accounts
2. Click Create Service Account
3. Fill in:

• Service account name: Opik Service Account
• ID: opik-sa
• Description: Service account for Opik role

4. Click Done

Step 3: Assign the Role to the Service Account

1. Go to IAM
2. Find the service account opik-sa@<my-project>.iam.gserviceaccount.com
3. Click the edit icon
4. Click Add Another Role > Select your custom role: Opik
5. Click Save

Step 4: Create and Download the Key

1. Go to Service Accounts
2. Click on the opik-sa account
3. Open the Keys tab
4. Click Add Key > Create new key
5. Choose JSON, click Create, and download the file

Open the downloaded JSON file, and copy its entire content to be used in the next step.

Final Step: Connect Opik to Vertex AI

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add Configuration”
3. Set:

• Provider: Vertex AI
• Location: Your model region (e.g., us-central1)
• Vertex AI API Key: Paste the full contents of the opik-key.json file here

4. Click Add configuration

Amazon Bedrock

Amazon Bedrock provides access to foundation models from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through AWS. Opik connects to Bedrock using the OpenAI Chat Completions API. Only models that support this API format will work with the Opik Playground. Check the supported models documentation to verify compatibility before configuring.

Prerequisites

Before configuring Bedrock in Opik, ensure you have:

1. An active AWS account with Bedrock access
2. Model access enabled for the models you want to use (see AWS documentation)
3. An API key or credentials configured for Bedrock access

Configuring Bedrock in Opik

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add Configuration”
3. Select “Bedrock” from the provider dropdown
4. Fill in the configuration:
   • Provider name: A unique identifier for this provider instance (e.g., “Bedrock us-east-1”)
   • URL: Your Bedrock endpoint URL (see format below)
   • Authentication: Your AWS Bedrock API key (see AWS documentation for setup instructions), or OAuth2 client credentials if your gateway issues short-lived tokens (see OAuth2 Client Credentials Authentication below)
   • Models list: Comma-separated list of models you want to use (e.g., us.anthropic.claude-sonnet-4-6,us.meta.llama3-2-3b-instruct-v1:0)
   • Custom headers (optional): Add any additional HTTP headers required by your configuration
5. Click Add configuration to save

Bedrock URL Format by Region

Bedrock endpoints follow this pattern: https://bedrock-runtime.<region>.amazonaws.com/openai/v1

Examples by Region:

• US East 1: https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1
• US West 2: https://bedrock-runtime.us-west-2.amazonaws.com/openai/v1
• Europe West 1 (Ireland): https://bedrock-runtime.eu-west-1.amazonaws.com/openai/v1
• Europe Central 1 (Frankfurt): https://bedrock-runtime.eu-central-1.amazonaws.com/openai/v1
• Asia Pacific (Tokyo): https://bedrock-runtime.ap-northeast-1.amazonaws.com/openai/v1
• Asia Pacific (Singapore): https://bedrock-runtime.ap-southeast-1.amazonaws.com/openai/v1

Multiple Bedrock Instances

You can configure multiple Bedrock providers for different AWS regions or accounts. Each instance appears separately in the provider dropdown, making it easy to switch between configurations in the Playground and Online Evaluation.

Ollama

Opik connects to Ollama using the OpenAI-compatible API, so you can use Ollama models for all LLM operations.

Configuring Ollama in Opik

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add configuration”
3. Select “Ollama” from the provider dropdown
4. Fill in:
   • Provider name: A name for this instance (e.g., “Ollama local”)
   • URL: Base URL of your Ollama instance, ending with /v1 (e.g., http://localhost:11434/v1)
   • API Key (optional): Leave blank unless your Ollama instance requires authentication
   • Click Connect: Opik checks that it can reach the instance and loads its model list
5. Click Save to store the configuration

You can configure multiple Ollama instances with different provider names and URLs.

vLLM / Custom Provider

Use this option to add any other OpenAI API-compliant provider such as vLLM, etc. You can configure multiple custom providers, each with their own unique name, URL, and models.

Configuration Steps

1. Provider Name: Enter a unique name to identify this custom provider (e.g., “vLLM Production”, “Ollama Local”, “Azure OpenAI Dev”)
2. URL: Enter your server URL, for example: http://host.docker.internal:8000/v1
3. Authentication (optional): If your model access requires authentication, either enter a static API key or use OAuth2 client credentials (see OAuth2 Client Credentials Authentication below). Otherwise, leave it blank.
4. Models: List all models available on your server. You’ll be able to select one of them for use later.
5. Custom Headers (optional): Add any additional HTTP headers required by your custom endpoint as key-value pairs.

# Helm values: harden a deployment whose providers are all public
component:
  backend:
    env:
      LLM_PROVIDER_DESTINATION_GUARD: "strict"

Accepted values are strict and relaxed (case-insensitive). When the variable is unset, the default for your launch path applies. Any other value, including an empty one, stops the backend at startup with a configuration error, so a typo never silently turns the check off. Docker Compose is the one exception for an empty value: it treats it as unset and uses its relaxed default.

The check applies to every LLM provider call except Vertex AI, including the built-in providers’ default endpoints and the free model. Vertex AI calls go to Google’s endpoint for the configured location through the Google client library, which this check doesn’t cover. With strict, a provider host that your DNS resolves to a private address, for example through a private endpoint or an egress proxy, is refused; use relaxed for such networks.

LLM_PROVIDER_DESTINATION_SCHEME sets which schemes strict mode accepts: plaintext_or_tls (the default on every launch path) allows http and https, as self-hosted model servers commonly run without TLS, and https_only allows https only. Values are case-insensitive, and any other value stops the backend at startup. relaxed mode doesn’t check the scheme.

Video URLs without a file extension get a HEAD request to detect their type. LLM_PROVIDER_VIDEO_URL_DESTINATION_GUARD applies the same destination check to that request (default relaxed on Docker Compose and Helm, strict on every other launch path); a refused URL is still sent to the model, just without the detected type. Set it to relaxed to detect the type of videos served from private networks.

Gemini and Vertex AI receive media as inline data, so Opik downloads the image, audio, video and PDF URLs in a message itself; only gs:// and data: URLs are passed through as they are. The same variable applies the destination check to each download. A media URL with another scheme, or one that is refused, redirects, or exceeds LLM_PROVIDER_MEDIA_DOWNLOAD_TIMEOUT (default 30s, at most 2m), fails the request with a 400, as do downloads that together exceed LLM_PROVIDER_MEDIA_DOWNLOAD_MAX_SIZE (default 20MB, at most 100MB, the limit for all media in one request). With relaxed, these downloads can reach any address the backend can, including your internal network, so use it only when everyone who can send messages or log traces is trusted with that access.

LLM_PROVIDER_MEDIA_URL_DESTINATION_SCHEME sets which schemes strict mode accepts for these downloads: plaintext_or_tls (the default on every launch path) allows http and https, and https_only allows https only. Values are case-insensitive, and any other value stops the backend at startup. relaxed mode doesn’t check it, but only http and https media URLs are ever downloaded.

Opik reads at most LLM_PROVIDER_ERROR_BODY_MAX_SIZE (default 64KB, at most 1MiB) of a custom provider’s error response and discards the rest; only a short message parsed from it is returned.

Provider calls are bounded on the backend by the settings below. The stream read timeout applies to every provider except Vertex AI. The thread and request limits apply to the streamed calls of the shared HTTP client, which every provider uses except Vertex AI and the OpenAI Responses API; a call that isn’t streamed runs on the thread that made it, outside these limits.

A value outside its range, or limits that contradict each other, stops the backend at startup with a configuration error.

Changing a provider’s URL requires re-entering its API key (or OAuth2 credentials), since stored secrets are only ever sent to the endpoint they were saved for.

Custom Headers

Some custom providers may require additional HTTP headers beyond the API key for authentication or routing purposes. You can configure these headers using the “Custom headers” section:

• Click ”+ Add header” to add a new header
• Enter the header name (e.g., X-Custom-Auth, X-Request-ID)
• Enter the header value
• Add multiple headers as needed
• Use the trash icon to remove headers

Each header needs a name, and names are compared ignoring case, so two headers can’t differ only in case (X-Team and x-team); saving such headers fails with a 400.

Common use cases for custom headers:

• Custom authentication: Additional authentication tokens or headers required by your infrastructure
• Request routing: Headers for routing requests to specific model versions or deployments
• Metadata tracking: Custom headers for tracking or logging purposes
• Enterprise features: Headers required for enterprise proxy configurations

Managing Multiple Custom Providers

Once you’ve configured multiple custom providers, you can:

• Edit any custom provider by selecting it from the provider dropdown in the configuration dialog
• Delete custom providers that are no longer needed
• Switch between different custom providers in the Playground and Automation Rules

Each custom provider appears as a separate option in the provider dropdown, making it easy to work with multiple self-hosted or custom LLM deployments.

OAuth2 Client Credentials Authentication

Some LLM gateways require a short-lived access token from an OAuth2 authorization server instead of a static API key. For these setups, custom providers (as well as Bedrock) can authenticate using the OAuth2 client credentials flow.

When configured, Opik requests an access token from your auth service using the client credentials grant, attaches it as a Bearer token to every model call, and caches and refreshes it automatically before it expires. If the provider rejects a token as invalid or expired (for example, after a revocation), Opik fetches a fresh token and retries the call.

Configuring OAuth2 Authentication

1. In the provider configuration dialog, switch the Authentication mode from Static API key to OAuth2 client credentials
2. Fill in:
   • Token URL: The token endpoint of your authorization server (e.g., https://auth.example.com/oauth/token)
   • Credentials: Enter your client_id and client_secret in the pre-filled rows. Add more rows if your auth service requires them (e.g., scope, audience). grant_type=client_credentials is added automatically by the dialog, so you don’t need to add it (REST API callers must include it themselves).
3. Click Test connection to verify the setup. Opik performs the token fetch on the backend and reports the token lifetime.
4. Click Save to store the configuration

A provider uses either a static API key or OAuth2 authentication, never both. Saving an OAuth2 configuration clears any stored static key, and switching back to a static key clears the stored OAuth2 configuration. Changing the Token URL requires re-entering the credentials, since stored secrets are only ever sent to the endpoint they were saved for.

The UI covers the standard OAuth2 client credentials flow. If your token endpoint deviates from the standard, additional options are available through the REST API in the auth_config field.

Credential Security

All provider credentials are encrypted at rest, whether a static API key or OAuth2 client credentials. Static API keys are shown masked after saving, and locked OAuth2 credential values are write-only.

Troubleshooting

• Authentication Errors: Ensure your API key is valid and hasn’t expired
• Access Denied: Check that your API key has the required permissions for the models you’re trying to use
• Rate Limiting: Adjust your request frequency or contact your provider to increase limits
• Token fetch failures: Use Test connection in the provider dialog; the error message includes the auth server’s response
• Missing token lifetime: The token reply must state its lifetime (expires_in by default); for endpoints that omit it, set fallback_ttl_seconds through the REST API