> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://www.comet.com/docs/opik/self-host/configure/large_csv_uploads/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://www.comet.com/_mcp/server.

# Large CSV uploads

# Enabling Large CSV Uploads

Opik accepts CSV file uploads of up to 2GB for dataset creation. For self-hosted deployments that need to process large CSV files, apply the additional configuration below.

## Overview

CSV upload supports:

* **CSV files up to 2GB** in size
* **Asynchronous processing** - files are processed in the background after upload

## Configuration Steps

### 1. Increase Idle Timeout

Large file uploads require more time to transfer. Increase the server idle timeout:

```bash
SERVER_IDLE_TIMEOUT: 10m
```

The default timeout is 30 seconds, which is insufficient for large file uploads. We recommend setting it to 10 minutes for files up to 2GB.

### 2. Configure Nginx (Kubernetes/Helm Deployments)

If you're using the Helm chart deployment, add the following configuration to your `values.yaml`:

```yaml
component:
  frontend:
    # Increase client body size limit to 2GB
    clientMaxBodySize: "2g"
    
    # Increase proxy timeouts for large file uploads
    upstreamConfig:
      proxy_read_timeout: 600s
      proxy_connect_timeout: 600s
      proxy_send_timeout: 600s
      client_max_body_size: 2g
```

### 3. Ensure Adequate Disk Space

The backend service temporarily buffers uploaded CSV files to disk before processing them. Ensure your backend pods/containers have:

* **Minimum 50GB of disk space** available
* **Sufficient IOPS** for concurrent file operations

### 4. Optional: Adjust Batch Size

You can optionally configure the batch size for CSV processing:

```bash
BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE: 1000
```

The default batch size is 1000 rows per batch. Adjust this based on your:

* Available memory
* Row complexity (number of columns, data size)
* Desired processing speed

## Docker Compose Deployments

For Docker Compose deployments, the configuration is slightly different:

### 1. Update docker-compose.yml

Add the environment variables to the backend service:

```yaml
services:
  backend:
    environment:
      - SERVER_IDLE_TIMEOUT=10m
      - BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE=1000
```

### 2. Update Nginx Configuration

The nginx configuration files already include the 2GB limit for local deployments. No additional changes are needed for `nginx_default_local.conf` or `nginx_local_be_local.conf`.

## Kubernetes/Helm Deployment Example

Here's a complete example for Helm chart deployments:

```yaml
# values.yaml
component:
  backend:
    env:
      SERVER_IDLE_TIMEOUT: "10m"
      BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE: "1000"
    
    # Ensure adequate disk space
    persistence:
      enabled: true
      size: 100Gi  # Adjust based on your needs
  
  frontend:
    clientMaxBodySize: "2g"
    
    upstreamConfig:
      proxy_read_timeout: 600s
      proxy_connect_timeout: 600s
      proxy_send_timeout: 600s
      client_max_body_size: 2g
```

Then upgrade your Helm release:

```bash
helm upgrade opik opik/opik -n opik -f values.yaml
```

## Verification

After applying the configuration:

1. **Restart services** to apply the changes
2. **Test with a small CSV** first (\< 100MB) to verify the feature works
3. **Monitor logs** during upload to ensure proper processing:

```bash
# Kubernetes
kubectl logs -n opik deployment/opik-backend -f | grep CSV

# Docker Compose
docker-compose logs -f backend | grep CSV
```

You should see log messages like:

```
CSV upload request for dataset 'xxx' on workspaceId 'xxx'
CSV upload accepted for dataset 'xxx' on workspaceId 'xxx', processing asynchronously
Starting asynchronous CSV processing for dataset 'xxx' on workspaceId 'xxx'
CSV processing completed for dataset 'xxx', total items: 'xxx'
```

## Troubleshooting

### Upload Fails with 413 Error

**Problem**: HTTP 413 Request Entity Too Large

**Solution**: Verify nginx configuration includes `client_max_body_size: 2g` at the server level, not just in location blocks.

### Upload Succeeds but Processing Fails

**Problem**: File uploads successfully but items don't appear in the dataset

**Solution**:

1. Check backend logs for processing errors
2. Verify adequate disk space is available
3. Check memory limits - large CSV files require sufficient memory for processing

### Timeout Errors

**Problem**: Upload times out before completing

**Solution**:

1. Increase `SERVER_IDLE_TIMEOUT` further (e.g., to 15m or 20m)
2. Increase nginx proxy timeouts in `upstreamConfig`
3. Check network bandwidth between client and server

### Out of Memory Errors

**Problem**: Backend service crashes or restarts during processing

**Solution**:

1. Reduce `BATCH_OPERATIONS_DATASETS_CSV_BATCH_SIZE` to process smaller batches
2. Increase backend service memory limits
3. Process smaller CSV files or split large files into multiple uploads

## Additional Resources

* [Scaling Opik](/self-host/scaling) - General scaling guidelines
* [Kubernetes Deployment](/self-host/kubernetes) - Helm chart documentation
* [Troubleshooting](/self-host/troubleshooting) - Common issues and solutions

> Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.