AI Providers

AI Providers let you connect LLMs for use in the Playground and Online Evaluation.

Using AI Providers

Once configured, providers appear in two places:

1. Playground — Test prompts interactively with different models.
2. Online Evaluation — Run LLM-as-a-judge scoring on your traces.

Adding a Provider

1. Click the Add configuration button in the top-right corner

2. In the Provider Configuration dialog that appears:
   • Select a provider from the dropdown menu
   • Enter your API key for that provider
   • Click Save to store the configuration

Supported Providers

Opik supports integration with various AI providers, including:

• OpenAI
• Anthropic
• OpenRouter
• Gemini
• VertexAI
• Azure OpenAI
• Amazon Bedrock
• Ollama (local or self-hosted, OpenAI-compatible)
• vLLM / any other OpenAI API-compliant provider

Provider-Specific Setup

Below are instructions for obtaining API keys and other required information for each supported provider:

OpenAI

1. Create or log in to your OpenAI account
2. Navigate to the API keys page
3. Click “Create new secret key”
4. Copy your API key (it will only be shown once)
5. In Opik, select “OpenAI” as the provider and paste your key

Anthropic

1. Sign up for or log in to Anthropic’s platform
2. Navigate to the API Keys page
3. Click “Create Key” and select the appropriate access level
4. Copy your API key (it will only be shown once)
5. In Opik, select “Anthropic” as the provider and paste your key

OpenRouter

1. Create or log in to your OpenRouter account
2. Navigate to the API Keys page
3. Create a new API key
4. Copy your API key
5. In Opik, select “OpenRouter” as the provider and paste your key

Gemini

1. Signup or login to Google AI Studio
2. Go to the API keys page\
3. Create a new API key for one your existing Google Cloud project
4. Copy your API key (it will only be shown once)
5. In Opik, select “Gemini” as the provider and paste your key

Azure OpenAI

Azure OpenAI provides enterprise-grade access to OpenAI models through Microsoft Azure. To use Azure OpenAI with Opik:

1. Get your Azure OpenAI endpoint URL

   • Go to portal.azure.com
   • Navigate to your Azure OpenAI resource
   • Copy your endpoint URL (it looks like https://your-company.openai.azure.com)

2. Construct the complete API URL

   • Add /openai/v1 to the end of your endpoint URL
   • Your complete URL should look like: https://your-company.openai.azure.com/openai/v1

3. Configure in Opik

   • In Opik, go to Workspace Settings > AI Providers
   • Click “Add Configuration”
   • Select “vLLM / Custom provider” from the dropdown
   • Enter your complete URL in the URL field: https://your-company.openai.azure.com/openai/v1
   • Add your Azure OpenAI API key in the API Key field
   • In the Models section, list the models you have deployed in Azure (e.g., gpt-4o)
   • Click Save to store the configuration

Once saved, you can use your Azure OpenAI models directly from Online Scores and the Playground.

Vertex AI

Option A: Setup via gcloud CLI

1. Create a Custom IAM Role

$
gcloud iam roles create opik \
>
  --project=<my-project> \
>
  --title="Opik" \
>
  --description="Custom IAM role for Opik" \
>
  --permissions=aiplatform.endpoints.predict,resourcemanager.projects.get \
>
  --stage=ALPHA

2. Create a Service Account

$
gcloud iam service-accounts create opik-sa \
>
  --description="Service account for Opik role" \
>
  --display-name="Opik Service Account"

3. Assign the Role to the Service Account

$
gcloud projects add-iam-policy-binding <my-project> \
>
  --member="serviceAccount:opik-sa@<my-project>.iam.gserviceaccount.com" \
>
  --role="projects/<my-project>/roles/opik"

4. Generate the Service Account Key File

$
gcloud iam service-accounts keys create opik-key.json \
>
  --iam-account=opik-sa@<my-project>.iam.gserviceaccount.com

The file opik-key.json contains your credentials. Open it in a text editor and copy the entire contents.

Option B: Setup via Google Cloud Console (UI)

Step 1: Create the Custom Role

1. Go to IAM > Roles
2. Click Create Role
3. Fill in the form:

• Title: Opik
• ID: opik
• Description: Custom IAM role for Opik
• Stage: Alpha

4. Click Add Permissions, then search for and add:

• aiplatform.endpoints.predict
• resourcemanager.projects.get

5. Click Create

Step 2: Create the Service Account

1. Go to IAM > Service Accounts
2. Click Create Service Account
3. Fill in:

• Service account name: Opik Service Account
• ID: opik-sa
• Description: Service account for Opik role

4. Click Done

Step 3: Assign the Role to the Service Account

1. Go to IAM
2. Find the service account opik-sa@<my-project>.iam.gserviceaccount.com
3. Click the edit icon
4. Click Add Another Role > Select your custom role: Opik
5. Click Save

Step 4: Create and Download the Key

1. Go to Service Accounts
2. Click on the opik-sa account
3. Open the Keys tab
4. Click Add Key > Create new key
5. Choose JSON, click Create, and download the file

Open the downloaded JSON file, and copy its entire content to be used in the next step.

Final Step: Connect Opik to Vertex AI

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add Configuration”
3. Set:

• Provider: Vertex AI
• Location: Your model region (e.g., us-central1)
• Vertex AI API Key: Paste the full contents of the opik-key.json file here

4. Click Add configuration

Amazon Bedrock

Amazon Bedrock provides access to foundation models from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through AWS. Opik connects to Bedrock using the OpenAI Chat Completions API. Only models that support this API format will work with the Opik Playground. Check the supported models documentation to verify compatibility before configuring.

Prerequisites

Before configuring Bedrock in Opik, ensure you have:

1. An active AWS account with Bedrock access
2. Model access enabled for the models you want to use (see AWS documentation)
3. An API key or credentials configured for Bedrock access

Configuring Bedrock in Opik

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add Configuration”
3. Select “Bedrock” from the provider dropdown
4. Fill in the configuration:
   • Provider name: A unique identifier for this provider instance (e.g., “Bedrock us-east-1”)
   • URL: Your Bedrock endpoint URL (see format below)
   • API Key: Your AWS Bedrock API key (see AWS documentation for setup instructions)
   • Models list: Comma-separated list of models you want to use (e.g., us.anthropic.claude-3-5-sonnet-20241022-v2:0,us.meta.llama3-2-3b-instruct-v1:0)
   • Custom headers (optional): Add any additional HTTP headers required by your configuration
5. Click Add configuration to save

Bedrock URL Format by Region

Bedrock endpoints follow this pattern: https://bedrock-runtime.<region>.amazonaws.com/openai/v1

Examples by Region:

• US East 1: https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1
• US West 2: https://bedrock-runtime.us-west-2.amazonaws.com/openai/v1
• Europe West 1 (Ireland): https://bedrock-runtime.eu-west-1.amazonaws.com/openai/v1
• Europe Central 1 (Frankfurt): https://bedrock-runtime.eu-central-1.amazonaws.com/openai/v1
• Asia Pacific (Tokyo): https://bedrock-runtime.ap-northeast-1.amazonaws.com/openai/v1
• Asia Pacific (Singapore): https://bedrock-runtime.ap-southeast-1.amazonaws.com/openai/v1

Multiple Bedrock Instances

You can configure multiple Bedrock providers for different AWS regions or accounts. Each instance appears separately in the provider dropdown, making it easy to switch between configurations in the Playground and Online Evaluation.

Ollama

Opik connects to Ollama using the OpenAI-compatible API, so you can use Ollama models for all LLM operations.

Configuring Ollama in Opik

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add configuration”
3. Select “Ollama” from the provider dropdown
4. Fill in:
   • Provider name: A name for this instance (e.g., “Ollama local”)
   • URL: Base URL of your Ollama instance, ending with /v1 (e.g., http://localhost:11434/v1)
   • API Key (optional): Leave blank unless your Ollama instance requires authentication
   • Use “Test connection” to verify Opik can reach the instance, then “Discover models” to load the model list
5. Click Save to store the configuration

You can configure multiple Ollama instances with different provider names and URLs.

vLLM / Custom Provider

Use this option to add any other OpenAI API-compliant provider such as vLLM, etc. You can configure multiple custom providers, each with their own unique name, URL, and models.

Configuration Steps

1. Provider Name: Enter a unique name to identify this custom provider (e.g., “vLLM Production”, “Ollama Local”, “Azure OpenAI Dev”)
2. URL: Enter your server URL, for example: http://host.docker.internal:8000/v1
3. API Key (optional): If your model access requires authentication, enter the API key. Otherwise, leave this field blank.
4. Models: List all models available on your server. You’ll be able to select one of them for use later.
5. Custom Headers (optional): Add any additional HTTP headers required by your custom endpoint as key-value pairs.

Custom Headers

Some custom providers may require additional HTTP headers beyond the API key for authentication or routing purposes. You can configure these headers using the “Custom headers” section:

• Click ”+ Add header” to add a new header
• Enter the header name (e.g., X-Custom-Auth, X-Request-ID)
• Enter the header value
• Add multiple headers as needed
• Use the trash icon to remove headers

Common use cases for custom headers:

• Custom authentication: Additional authentication tokens or headers required by your infrastructure
• Request routing: Headers for routing requests to specific model versions or deployments
• Metadata tracking: Custom headers for tracking or logging purposes
• Enterprise features: Headers required for enterprise proxy configurations

Query Parameters

Some gateways require specific query parameters on every request. For example, Azure OpenAI often requires an api-version parameter.

You can use the “Query parameters” section to define these key-value pairs, and Opik will automatically append them to every outgoing request.

• Click ”+ Add query parameter” to add a new pair
• Enter the parameter name (e.g., api-version)
• Enter the parameter value (e.g., 2024-08-01-preview)
• Add as many parameters as needed
• Use the trash icon to remove parameters

Parameters are automatically URL-encoded and appended with ? or & as appropriate. Leave this section empty for providers that don’t require extra query parameters.

Auth Header Name and Suppressing the Default Authorization Header

Most OpenAI-compatible providers expect an Authorization: Bearer <key> header — Opik sends this automatically. Some gateways instead expect the key in a differently named header (such as api-key or a tenant-specific name), sometimes without any Authorization header at all.

• Auth header name (optional): If set, Opik sends the API key as {name}: <key> in addition to the Authorization: Bearer <key> header. Most gateways ignore the extra Authorization header, so this alone is often enough.
• Suppress default Authorization header (toggle, default off): Enable this only if your gateway explicitly rejects requests containing Authorization: Bearer. When on, Opik sends only the custom header configured above.

URL Templates with {model}

When a gateway expects the model or deployment name in the URL path rather than in the request body, you can write the URL once using {model} as a placeholder. This avoids creating one provider entry per model:

https://my-gateway.example.com/deployments/{model}

At request time Opik substitutes {model} with the name of whichever model the end-user picked from the Models list. The Models field then contains all supported names (e.g. gpt-4o, gpt-4o-mini, llama-3.1-70b), and the playground / Online Evaluation dropdown shows each one as a selectable model routed to its own upstream path.

If your URL does not contain {model}, the substitution step is skipped and the URL is used verbatim — existing configurations are unaffected.

The model name is interpolated raw (no URL encoding) so it lands as a literal path segment, matching what most gateways expect. Use model names containing only alphanumeric characters and hyphens. Values containing / (e.g. HuggingFace-style mistralai/Mistral-7B-Instruct-v0.3) will split into extra path segments and Opik will emit a warning in the backend logs at request time.

Azure OpenAI via API Management gateway

Many enterprise customers front Azure OpenAI with Azure API Management (APIM). APIM typically requires:

• A deployment-specific URL path (/openai/deployments/{deployment}/chat/completions), or an organization-specific variant such as /za/deployments/{deployment}/chat/completions.
• A mandatory api-version query parameter.
• An api-key header (or a tenant-specific header name) instead of Authorization: Bearer.

Configure it as follows:

1. Select vLLM / Custom provider.
2. Provider name: something descriptive (e.g. azure-apim-prod).
3. URL: full gateway path, including the deployment placeholder, e.g. https://my-gateway.azure-api.net/openai/deployments/{model}.
4. API key: your APIM subscription key (or Azure OpenAI key, depending on gateway configuration).
5. Models list: comma-separated deployment names, e.g. gpt-4o, gpt-4o-mini-ZA. These will be substituted into {model} at request time.
6. Query parameters: add api-version with the value required by your gateway (e.g. 2024-08-01-preview).
7. Auth header name: set to api-key (or your tenant-specific header name).
8. Suppress default Authorization header: turn on only if your APIM policy rejects requests that include Authorization: Bearer. Most policies ignore the extra header, so leave it off unless you’ve confirmed it’s rejected.

Managing Multiple Custom Providers

Once you’ve configured multiple custom providers, you can:

• Edit any custom provider by selecting it from the provider dropdown in the configuration dialog
• Delete custom providers that are no longer needed
• Switch between different custom providers in the Playground and Automation Rules

Each custom provider appears as a separate option in the provider dropdown, making it easy to work with multiple self-hosted or custom LLM deployments.

API Key Security

API keys are encrypted and stored securely. Only the name and provider type are visible after configuration.

Troubleshooting

• Authentication Errors: Ensure your API key is valid and hasn’t expired
• Access Denied: Check that your API key has the required permissions for the models you’re trying to use
• Rate Limiting: Adjust your request frequency or contact your provider to increase limits