Configuration

AI Providers

The AI Providers tab allows you to configure connections to different Large Language Models (LLMs). This page explains how to set up and manage AI provider integrations within Opik.

Overview

Connecting AI providers enables you to:

  • Send prompts and receive responses from different LLMs
  • Set up a provider in one place and use it across projects
  • Automatically record model metadata in the Playground
  • Track and analyze traces using online evaluation rules

Managing AI Providers

Viewing Existing Providers

AI Providers Table

The AI Providers page displays a table of all configured connections with the following columns:

  • Name: The name or identifier of the API key
  • Created: The date and time when the provider was configured
  • Provider: The type of AI provider (e.g., OpenAI)

Adding a New Provider Configuration

To add a new AI provider:

  1. Click the Add configuration button in the top-right corner
New AI Provider Modal
  1. In the Provider Configuration dialog that appears:
    • Select a provider from the dropdown menu
    • Enter your API key for that provider
    • Click Save to store the configuration

Supported Providers

Opik supports integration with various AI providers, including:

  • OpenAI
  • Anthropic
  • OpenRouter
  • Gemini
  • VertexAI
  • LM Studio (coming soon)
  • vLLM / Ollama / any other OpenAI API-compliant provider

If you would like us to support additional LLM providers, please let us know by opening an issue on GitHub.

Provider-Specific Setup

Below are instructions for obtaining API keys and other required information for each supported provider:

OpenAI

  1. Create or log in to your OpenAI account
  2. Navigate to the API keys page
  3. Click “Create new secret key”
  4. Copy your API key (it will only be shown once)
  5. In Opik, select “OpenAI” as the provider and paste your key

Anthropic

  1. Sign up for or log in to Anthropic’s platform
  2. Navigate to the API Keys page
  3. Click “Create Key” and select the appropriate access level
  4. Copy your API key (it will only be shown once)
  5. In Opik, select “Anthropic” as the provider and paste your key

OpenRouter

  1. Create or log in to your OpenRouter account
  2. Navigate to the API Keys page
  3. Create a new API key
  4. Copy your API key
  5. In Opik, select “OpenRouter” as the provider and paste your key

Gemini

  1. Signup or login to Google AI Studio
  2. Go to the API keys page\
  3. Create a new API key for one your existing Google Cloud project
  4. Copy your API key (it will only be shown once)
  5. In Opik, select “Gemini” as the provider and paste your key

Vertex AI

Option A: Setup via gcloud CLI
  1. Create a Custom IAM Role
$gcloud iam roles create opik \
>  --project=<my-project> \
>  --title="Opik" \
>  --description="Custom IAM role for Opik" \
>  --permissions=aiplatform.endpoints.predict,resourcemanager.projects.get \
>  --stage=ALPHA
  1. Create a Service Account
$gcloud iam service-accounts create opik-sa \
>  --description="Service account for Opik role" \
>  --display-name="Opik Service Account"
  1. Assign the Role to the Service Account
$gcloud projects add-iam-policy-binding <my-project> \
>  --member="serviceAccount:opik-sa@<my-project>.iam.gserviceaccount.com" \
>  --role="projects/<my-project>/roles/opik"
  1. Generate the Service Account Key File
$gcloud iam service-accounts keys create opik-key.json \
>  --iam-account=opik-sa@<my-project>.iam.gserviceaccount.com

The file opik-key.json contains your credentials. Open it in a text editor and copy the entire contents.

Option B: Setup via Google Cloud Console (UI)

Step 1: Create the Custom Role

  1. Go to IAM → Roles
  2. Click Create Role
  3. Fill in the form:
  • Title: Opik
  • ID: opik
  • Description: Custom IAM role for Opik
  • Stage: Alpha
  1. Click Add Permissions, then search for and add:
  • aiplatform.endpoints.predict
  • resourcemanager.projects.get
  1. Click Create

Step 2: Create the Service Account

  1. Go to IAM → Service Accounts
  2. Click Create Service Account
  3. Fill in:
  • Service account name: Opik Service Account
  • ID: opik-sa
  • Description: Service account for Opik role
  1. Click Done

Step 3: Assign the Role to the Service Account

  1. Go to IAM
  2. Find the service account opik-sa@<my-project>.iam.gserviceaccount.com
  3. Click the edit icon
  4. Click Add Another Role → Select your custom role: Opik
  5. Click Save

Step 4: Create and Download the Key

  1. Go to Service Accounts
  2. Click on the opik-sa account
  3. Open the Keys tab
  4. Click Add Key → Create new key
  5. Choose JSON, click Create, and download the file

Open the downloaded JSON file, and copy its entire content to be used in the next step.

Final Step: Connect Opik to Vertex AI
  1. In Opik, go to Configuration → AI Providers
  2. Click “Add Configuration”
  3. Set:
  • Provider: Vertex AI
  • Location: Your model region (e.g., us-central1)
  • Vertex AI API Key: Paste the full contents of the opik-key.json file here
  1. Click Add configuration

vLLM / Custom Provider

Use this option to add any OpenAI API-compliant provider such as vLLM, Ollama, etc.

New Custom AI Provider Modal

For the URL section, enter your server URL, for example: http://host.docker.internal:8000/v1.

If you’re running Opik locally, you would need to use http://host.docker.internal:<PORT>/v1 for Mac and Windows or http://172.17.0.1:<PORT>/v1 for Linux, and not http://localhost.

If your model access requires authentication via an API key, enter it in the API Key field. Otherwise, you can leave this field blank.

In the Models section, list all models available on your server. You’ll be able to select one of them for use later.

API Key Security

API keys are securely stored and encrypted in the Opik system. Only the name and provider type are visible in the interface after configuration. The actual key values are not displayed after initial setup.

Using AI Providers

Once configured, AI providers can be used in:

  1. Playground: For interactive experimentation with different models
  2. Online Evaluation: For systematic evaluation of model performance

Best Practices

  • Use descriptive names for your API keys to easily identify their purpose
  • Regularly rotate API keys according to your organization’s security policies
  • Create separate keys for development and production environments
  • Monitor usage to manage costs effectively

Troubleshooting

Common Issues

  • Authentication Errors: Ensure your API key is valid and hasn’t expired
  • Access Denied: Check that your API key has the required permissions for the models you’re trying to use
  • Rate Limiting: If you’re encountering rate limits, consider adjusting your request frequency or contacting your provider to increase your limits

Additional Resources

  • For programmatic configuration of AI providers, see the API Reference
  • To learn about using different models in your application, see the SDK Documentation