AI Providers

AI Providers let you connect LLMs for use in the Playground and Online Evaluation.

Using AI Providers

Once configured, providers appear in two places:

1. Playground — Test prompts interactively with different models.
2. Online Evaluation — Run LLM-as-a-judge scoring on your traces.

Adding a Provider

1. Click the Add configuration button in the top-right corner

2. In the Provider Configuration dialog that appears:
   • Select a provider
   • Enter your API key for that provider
   • Click Save to store the configuration

Supported Providers

Opik supports integration with various AI providers, including:

• OpenAI
• Anthropic
• OpenRouter
• Gemini
• VertexAI
• Azure OpenAI
• Amazon Bedrock
• Ollama (local or self-hosted, OpenAI-compatible)
• vLLM / any other OpenAI API-compliant provider

Provider-Specific Setup

Below are instructions for obtaining API keys and other required information for each supported provider:

OpenAI

1. Create or log in to your OpenAI account
2. Navigate to the API keys page
3. Click “Create new secret key”
4. Copy your API key (it will only be shown once)
5. In Opik, select “OpenAI” as the provider and paste your key

Anthropic

1. Sign up for or log in to Anthropic’s platform
2. Navigate to the API Keys page
3. Click “Create Key” and select the appropriate access level
4. Copy your API key (it will only be shown once)
5. In Opik, select “Anthropic” as the provider and paste your key

OpenRouter

1. Create or log in to your OpenRouter account
2. Navigate to the API Keys page
3. Create a new API key
4. Copy your API key
5. In Opik, select “OpenRouter” as the provider and paste your key

Gemini

1. Signup or login to Google AI Studio
2. Go to the API keys page\
3. Create a new API key for one your existing Google Cloud project
4. Copy your API key (it will only be shown once)
5. In Opik, select “Gemini” as the provider and paste your key

Azure OpenAI

Azure OpenAI provides enterprise-grade access to OpenAI models through Microsoft Azure. To use Azure OpenAI with Opik:

1. Get your Azure OpenAI endpoint URL

   • Go to portal.azure.com
   • Navigate to your Azure OpenAI resource
   • Copy your endpoint URL (it looks like https://your-company.openai.azure.com)

2. Construct the complete API URL

   • Add /openai/v1 to the end of your endpoint URL
   • Your complete URL should look like: https://your-company.openai.azure.com/openai/v1

3. Configure in Opik

   • In Opik, go to Workspace Settings > AI Providers
   • Click “Add Configuration”
   • Select “vLLM / Custom provider” from the dropdown
   • Enter your complete URL in the URL field: https://your-company.openai.azure.com/openai/v1
   • Add your Azure OpenAI API key in the API Key field
   • In the Models section, list the models you have deployed in Azure (e.g., gpt-4o)
   • Click Save to store the configuration

Once saved, you can use your Azure OpenAI models directly from Online Scores and the Playground.

Vertex AI

Option A: Setup via gcloud CLI

1. Create a Custom IAM Role

gcloud iam roles create opik \
  --project=<my-project> \
  --title="Opik" \
  --description="Custom IAM role for Opik" \
  --permissions=aiplatform.endpoints.predict,resourcemanager.projects.get \
  --stage=ALPHA

2. Create a Service Account

gcloud iam service-accounts create opik-sa \
  --description="Service account for Opik role" \
  --display-name="Opik Service Account"

3. Assign the Role to the Service Account

gcloud projects add-iam-policy-binding <my-project> \
  --member="serviceAccount:opik-sa@<my-project>.iam.gserviceaccount.com" \
  --role="projects/<my-project>/roles/opik"

4. Generate the Service Account Key File

gcloud iam service-accounts keys create opik-key.json \
  --iam-account=opik-sa@<my-project>.iam.gserviceaccount.com

The file opik-key.json contains your credentials. Open it in a text editor and copy the entire contents.

Option B: Setup via Google Cloud Console (UI)

Step 1: Create the Custom Role

1. Go to IAM > Roles
2. Click Create Role
3. Fill in the form:

• Title: Opik
• ID: opik
• Description: Custom IAM role for Opik
• Stage: Alpha

4. Click Add Permissions, then search for and add:

• aiplatform.endpoints.predict
• resourcemanager.projects.get

5. Click Create

Step 2: Create the Service Account

1. Go to IAM > Service Accounts
2. Click Create Service Account
3. Fill in:

• Service account name: Opik Service Account
• ID: opik-sa
• Description: Service account for Opik role

4. Click Done

Step 3: Assign the Role to the Service Account

1. Go to IAM
2. Find the service account opik-sa@<my-project>.iam.gserviceaccount.com
3. Click the edit icon
4. Click Add Another Role > Select your custom role: Opik
5. Click Save

Step 4: Create and Download the Key

1. Go to Service Accounts
2. Click on the opik-sa account
3. Open the Keys tab
4. Click Add Key > Create new key
5. Choose JSON, click Create, and download the file

Open the downloaded JSON file, and copy its entire content to be used in the next step.

Final Step: Connect Opik to Vertex AI

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add Configuration”
3. Set:

• Provider: Vertex AI
• Location: Your model region (e.g., us-central1)
• Vertex AI API Key: Paste the full contents of the opik-key.json file here

4. Click Add configuration

Amazon Bedrock

Amazon Bedrock provides access to foundation models from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through AWS. Opik connects to Bedrock using the OpenAI Chat Completions API. Only models that support this API format will work with the Opik Playground. Check the supported models documentation to verify compatibility before configuring.

Prerequisites

Before configuring Bedrock in Opik, ensure you have:

1. An active AWS account with Bedrock access
2. Model access enabled for the models you want to use (see AWS documentation)
3. An API key or credentials configured for Bedrock access

Configuring Bedrock in Opik

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add Configuration”
3. Select “Bedrock” from the provider dropdown
4. Fill in the configuration:
   • Provider name: A unique identifier for this provider instance (e.g., “Bedrock us-east-1”)
   • URL: Your Bedrock endpoint URL (see format below)
   • Authentication: Your AWS Bedrock API key (see AWS documentation for setup instructions), or OAuth2 client credentials if your gateway issues short-lived tokens (see OAuth2 Client Credentials Authentication below)
   • Models list: Comma-separated list of models you want to use (e.g., us.anthropic.claude-3-5-sonnet-20241022-v2:0,us.meta.llama3-2-3b-instruct-v1:0)
   • Custom headers (optional): Add any additional HTTP headers required by your configuration
5. Click Add configuration to save

Bedrock URL Format by Region

Bedrock endpoints follow this pattern: https://bedrock-runtime.<region>.amazonaws.com/openai/v1

Examples by Region:

• US East 1: https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1
• US West 2: https://bedrock-runtime.us-west-2.amazonaws.com/openai/v1
• Europe West 1 (Ireland): https://bedrock-runtime.eu-west-1.amazonaws.com/openai/v1
• Europe Central 1 (Frankfurt): https://bedrock-runtime.eu-central-1.amazonaws.com/openai/v1
• Asia Pacific (Tokyo): https://bedrock-runtime.ap-northeast-1.amazonaws.com/openai/v1
• Asia Pacific (Singapore): https://bedrock-runtime.ap-southeast-1.amazonaws.com/openai/v1

Multiple Bedrock Instances

You can configure multiple Bedrock providers for different AWS regions or accounts. Each instance appears separately in the provider dropdown, making it easy to switch between configurations in the Playground and Online Evaluation.

Ollama

Opik connects to Ollama using the OpenAI-compatible API, so you can use Ollama models for all LLM operations.

Configuring Ollama in Opik

1. In Opik, go to Workspace Settings > AI Providers
2. Click “Add configuration”
3. Select “Ollama” from the provider dropdown
4. Fill in:
   • Provider name: A name for this instance (e.g., “Ollama local”)
   • URL: Base URL of your Ollama instance, ending with /v1 (e.g., http://localhost:11434/v1)
   • API Key (optional): Leave blank unless your Ollama instance requires authentication
   • Use “Test connection” to verify Opik can reach the instance, then “Discover models” to load the model list
5. Click Save to store the configuration

You can configure multiple Ollama instances with different provider names and URLs.

vLLM / Custom Provider

Use this option to add any other OpenAI API-compliant provider such as vLLM, etc. You can configure multiple custom providers, each with their own unique name, URL, and models.

Configuration Steps

1. Provider Name: Enter a unique name to identify this custom provider (e.g., “vLLM Production”, “Ollama Local”, “Azure OpenAI Dev”)
2. URL: Enter your server URL, for example: http://host.docker.internal:8000/v1
3. Authentication (optional): If your model access requires authentication, either enter a static API key or use OAuth2 client credentials (see OAuth2 Client Credentials Authentication below). Otherwise, leave it blank.
4. Models: List all models available on your server. You’ll be able to select one of them for use later.
5. Custom Headers (optional): Add any additional HTTP headers required by your custom endpoint as key-value pairs.

Custom Headers

Some custom providers may require additional HTTP headers beyond the API key for authentication or routing purposes. You can configure these headers using the “Custom headers” section:

• Click ”+ Add header” to add a new header
• Enter the header name (e.g., X-Custom-Auth, X-Request-ID)
• Enter the header value
• Add multiple headers as needed
• Use the trash icon to remove headers

Common use cases for custom headers:

• Custom authentication: Additional authentication tokens or headers required by your infrastructure
• Request routing: Headers for routing requests to specific model versions or deployments
• Metadata tracking: Custom headers for tracking or logging purposes
• Enterprise features: Headers required for enterprise proxy configurations

Managing Multiple Custom Providers

Once you’ve configured multiple custom providers, you can:

• Edit any custom provider by selecting it from the provider dropdown in the configuration dialog
• Delete custom providers that are no longer needed
• Switch between different custom providers in the Playground and Automation Rules

Each custom provider appears as a separate option in the provider dropdown, making it easy to work with multiple self-hosted or custom LLM deployments.

OAuth2 Client Credentials Authentication

Some LLM gateways require a short-lived access token from an OAuth2 authorization server instead of a static API key. For these setups, custom providers (as well as Bedrock) can authenticate using the OAuth2 client credentials flow.

When configured, Opik requests an access token from your auth service using the client credentials grant, attaches it as a Bearer token to every model call, and caches and refreshes it automatically before it expires. If the provider rejects a token as invalid or expired (for example, after a revocation), Opik fetches a fresh token and retries the call.

Configuring OAuth2 Authentication

1. In the provider configuration dialog, switch the Authentication mode from Static API key to OAuth2 client credentials
2. Fill in:
   • Token URL: The token endpoint of your authorization server (e.g., https://auth.example.com/oauth/token)
   • Credentials: Enter your client_id and client_secret in the pre-filled rows. Add more rows if your auth service requires them (e.g., scope, audience). grant_type=client_credentials is added automatically by the dialog, so you don’t need to add it (REST API callers must include it themselves).
3. Click Test connection to verify the setup. Opik performs the token fetch on the backend and reports the token lifetime.
4. Click Save to store the configuration

A provider uses either a static API key or OAuth2 authentication, never both. Saving an OAuth2 configuration clears any stored static key, and switching back to a static key clears the stored OAuth2 configuration. Changing the Token URL requires re-entering the credentials, since stored secrets are only ever sent to the endpoint they were saved for.

The UI covers the standard OAuth2 client credentials flow. If your token endpoint deviates from the standard, additional options are available through the REST API in the auth_config field.

Credential Security

All provider credentials are encrypted at rest, whether a static API key or OAuth2 client credentials. Static API keys are shown masked after saving, and locked OAuth2 credential values are write-only.

Troubleshooting

• Authentication Errors: Ensure your API key is valid and hasn’t expired
• Access Denied: Check that your API key has the required permissions for the models you’re trying to use
• Rate Limiting: Adjust your request frequency or contact your provider to increase limits
• Token fetch failures: Use Test connection in the provider dialog; the error message includes the auth server’s response
• Missing token lifetime: The token reply must state its lifetime (expires_in by default); for endpoints that omit it, set fallback_ttl_seconds through the REST API