Configuration

AI Providers

The AI Providers tab allows you to configure connections to different Large Language Models (LLMs). This page explains how to set up and manage AI provider integrations within Opik.

Overview

Connecting AI providers enables you to:

  • Send prompts and receive responses from different LLMs
  • Set up a provider in one place and use it across projects
  • Automatically record model metadata in the Playground
  • Track and analyze traces using online evaluation rules

Managing AI Providers

Viewing Existing Providers

AI Providers Table

The AI Providers page displays a table of all configured connections with the following columns:

  • Name: The name or identifier of the API key
  • Created: The date and time when the provider was configured
  • Provider: The type of AI provider (e.g., OpenAI)

Adding a New Provider Configuration

To add a new AI provider:

  1. Click the Add configuration button in the top-right corner
New AI Provider Modal
  1. In the Provider Configuration dialog that appears:
    • Select a provider from the dropdown menu
    • Enter your API key for that provider
    • Click Save to store the configuration

Supported Providers

Opik supports integration with various AI providers, including:

  • OpenAI
  • Anthropic
  • OpenRouter
  • Ollama (experimental)
  • Gemini
  • VertexAI
  • LM Studio (coming soon)

If you would like us to support additional LLM providers, please let us know by opening an issue on GitHub.

Provider-Specific Setup

Below are instructions for obtaining API keys and other required information for each supported provider:

OpenAI

  1. Create or log in to your OpenAI account
  2. Navigate to the API keys page
  3. Click “Create new secret key”
  4. Copy your API key (it will only be shown once)
  5. In Opik, select “OpenAI” as the provider and paste your key

Anthropic

  1. Sign up for or log in to Anthropic’s platform
  2. Navigate to the API Keys page
  3. Click “Create Key” and select the appropriate access level
  4. Copy your API key (it will only be shown once)
  5. In Opik, select “Anthropic” as the provider and paste your key

OpenRouter

  1. Create or log in to your OpenRouter account
  2. Navigate to the API Keys page
  3. Create a new API key
  4. Copy your API key
  5. In Opik, select “OpenRouter” as the provider and paste your key

Ollama

If you are using Ollama, you will need to ensure that Ollama’s security configuration is set up correctly to avoid CORS issues. If you are running Ollama in production, we recommend reviewing the Ollama documentation for advice on best practices.

If you are simply looking at using Ollama in the Oplik playground, we have released a utility to help you access Ollama from your browser. The Python SDK includes a simple reverse proxy that you can run on your local machine to proxy requests to Ollama:

1

Install the Python SDK

$# Install the latest version of the SDK
>pip install opik -U
2

Run the reverse proxy

$# Run the reverse proxy
>opik proxy --ollama

Once the server is running, you should see the output:

3

You can then configure the AI providers using the URL and models parameters returned by the proxy server in the console:

Don’t forget to update the model list with the models supported by your proxy server!

You will need to keep the proxy server running for the playground to work. If it is not running, you will see the error: Unexpected error.

Gemini

  1. Signup or login to Google AI Studio
  2. Go to the API keys page\
  3. Create a new API key for one your existing Google Cloud project
  4. Copy your API key (it will only be shown once)
  5. In Opik, select “Gemini” as the provider and paste your key

Vertex AI

Option A: Setup via gcloud CLI
  1. Create a Custom IAM Role
$gcloud iam roles create opik \
>  --project=<my-project> \
>  --title="Opik" \
>  --description="Custom IAM role for Opik" \
>  --permissions=aiplatform.endpoints.predict,resourcemanager.projects.get \
>  --stage=ALPHA
  1. Create a Service Account
$gcloud iam service-accounts create opik-sa \
>  --description="Service account for Opik role" \
>  --display-name="Opik Service Account"
  1. Assign the Role to the Service Account
$gcloud projects add-iam-policy-binding <my-project> \
>  --member="serviceAccount:opik-sa@<my-project>.iam.gserviceaccount.com" \
>  --role="projects/<my-project>/roles/opik"
  1. Generate the Service Account Key File
$gcloud iam service-accounts keys create opik-key.json \
>  --iam-account=opik-sa@<my-project>.iam.gserviceaccount.com

The file opik-key.json contains your credentials. Open it in a text editor and copy the entire contents.

Option B: Setup via Google Cloud Console (UI)

Step 1: Create the Custom Role

  1. Go to IAM → Roles
  2. Click Create Role
  3. Fill in the form:
  • Title: Opik
  • ID: opik
  • Description: Custom IAM role for Opik
  • Stage: Alpha
  1. Click Add Permissions, then search for and add:
  • aiplatform.endpoints.predict
  • resourcemanager.projects.get
  1. Click Create

Step 2: Create the Service Account

  1. Go to IAM → Service Accounts
  2. Click Create Service Account
  3. Fill in:
  • Service account name: Opik Service Account
  • ID: opik-sa
  • Description: Service account for Opik role
  1. Click Done

Step 3: Assign the Role to the Service Account

  1. Go to IAM
  2. Find the service account opik-sa@<my-project>.iam.gserviceaccount.com
  3. Click the edit icon
  4. Click Add Another Role → Select your custom role: Opik
  5. Click Save

Step 4: Create and Download the Key

  1. Go to Service Accounts
  2. Click on the opik-sa account
  3. Open the Keys tab
  4. Click Add Key → Create new key
  5. Choose JSON, click Create, and download the file

Open the downloaded JSON file, and copy its entire content to be used in the next step.

Final Step: Connect Opik to Vertex AI
  1. In Opik, go to Configuration → AI Providers
  2. Click “Add Configuration”
  3. Set:
  • Provider: Vertex AI
  • Location: Your model region (e.g., us-central1)
  • Vertex AI API Key: Paste the full contents of the opik-key.json file here
  1. Click Add configuration

API Key Security

API keys are securely stored and encrypted in the Opik system. Only the name and provider type are visible in the interface after configuration. The actual key values are not displayed after initial setup.

Using AI Providers

Once configured, AI providers can be used in:

  1. Playground: For interactive experimentation with different models
  2. Online Evaluation: For systematic evaluation of model performance

Best Practices

  • Use descriptive names for your API keys to easily identify their purpose
  • Regularly rotate API keys according to your organization’s security policies
  • Create separate keys for development and production environments
  • Monitor usage to manage costs effectively

Troubleshooting

Common Issues

  • Authentication Errors: Ensure your API key is valid and hasn’t expired
  • Access Denied: Check that your API key has the required permissions for the models you’re trying to use
  • Rate Limiting: If you’re encountering rate limits, consider adjusting your request frequency or contacting your provider to increase your limits

Additional Resources

  • For programmatic configuration of AI providers, see the API Reference
  • To learn about using different models in your application, see the SDK Documentation