{"id":20929,"date":"2026-11-17T00:14:43","date_gmt":"2026-11-17T00:14:43","guid":{"rendered":"https:\/\/www.comet.com\/site\/?p=20929"},"modified":"2026-11-17T00:15:48","modified_gmt":"2026-11-17T00:15:48","slug":"improve-opik-ingestion-30x","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/","title":{"rendered":"How we made Opik dataset and experiment ingestion 30\u00d7 faster"},"content":{"rendered":"<p><a href=\"https:\/\/www.comet.com\/site\/products\/opik\/\">Opik<\/a> is Comet\u2019s open-source platform for tracing, evaluating, testing, and monitoring LLM and agent applications. It tracks every call an application makes, including prompts, tool calls, retrievals, and LLM outputs. This gives teams tools to understand the application\u2019s behavior, find where it goes wrong, and identify what to improve, from the developer workstation to production.<\/p>\n<p>LLM and agent applications are hard to test, mainly because they are not deterministic. The same input can produce different outputs from one run to the next, and changing a prompt, a model, or a tool can have unintended effects that break expected system behavior. To reliably know whether a change caused the intended behavior, run the application over a fixed set of cases and compare the results with previous runs. In Opik, this kind of set is called a dataset or test suite. These are versioned collections of inputs and often their expected outputs. In the case of test suites, they go one step further, their items state the expected behavior as natural-language assertions, which an <a href=\"https:\/\/www.comet.com\/site\/blog\/llm-as-a-judge\/\">LLM judge<\/a> checks against the actual outputs.<\/p>\n<p>An experiment is a run of the application over a dataset or test suite that records every item produced and its scores. This lets us compare runs item by item and see the full state of the application.<\/p>\n<p>Every evaluation runs the application on each dataset item and writes an experiment item with its trace and scores, which then are read back to compare runs. All this data, the dataset items and the experiment results, lands in ClickHouse, our analytics database. So how fast an evaluation runs depends both on how fast the application produces its outputs and on how fast Opik can write and read that data.<\/p>\n<p>Running periodic evaluations on large datasets wasn&#8217;t practical in Opik: a large dataset upload could take close to an hour, and we couldn&#8217;t run experiments on such datasets in a reasonable time. Once our engineering team figured that out, we set out to fix that, so teams with large datasets could run evaluations in Opik as often as they need.<\/p>\n<p>We measured the whole dataset and experiment pipeline to learn where the time and resources went and how far each setup could go. The resulting changes made dataset uploads up to ~30\u00d7 faster and experiments ~20\u00d7 faster, all while using far less memory than before.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-thiago-4.png\" alt=\"\" \/><\/p>\n<h2>The problem: periodic evaluation at scale<\/h2>\n<p>Periodic evaluations repeat the same cycles on every run. Each run uploads or refreshes the datasets\/test suites, runs the experiment, and writes its results, then reads them back to compare. At around 100K items, every step of the cycle was too slow to repeat, and the first step often didn\u2019t finish in a reasonable time. For testing, we defined a dataset with 100K relatively small items, and a heavier one with 30K but much larger items.<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Step in the cycle<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Where we started<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Why it blocked periodic evaluations<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Upload the dataset<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>About 40 minutes and 17.8 GiB RAM for 30K large rows (SDK 2.2.44)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>40 minutes per run, and memory grew with the dataset<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Upload a larger dataset<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>The full 100K small rows needed ~72 GB RAM, many times the size of the data itself<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>It could not run on a 32 GiB machine, let alone a laptop or CI runner<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Write the experiment<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>5 minutes for 100K items, single-threaded<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Five minutes with the client idle three-quarters of the time<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Read the results back<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4 min 33 s for 100K items (SDK 2.2.74); HTTP 500 when read concurrently<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Over four minutes, just to look at results<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>Our starting point was to test scenarios representative of the workloads we were targeting and measure how the system would behave in each one before changing anything.<\/p>\n<\/p>\n<p>Overall latency was a crucial KPI, but not the only one. A change that is faster but holds the dataset in memory, sends more bytes, or fails less clearly is not a win for someone running evaluations on a schedule or on their CI. So every scenario recorded KPIs on both sides of the wire. On the client, we measured:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>What we measured<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>How<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Why it mattered<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Latency<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Wall time and items\/s per run, plus request p50<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>The headline: how long one step of the cycle takes<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Client memory<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Peak RSS of the uploading process<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Decides whether a large upload runs at all on a laptop or CI runner<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Client CPU<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>CPU ms per item<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Shows whether a change removed work or only spread it over more threads<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Efficiency<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Requests in flight against the thread count; time waiting versus working<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Shows whether the client was idle or the bottleneck<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Bytes sent<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Wire bytes and request count<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Rules out &#8220;faster because it sends less&#8221; and tracks egress cost<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Developer experience<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Error handling and validation behavior<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>A faster upload must still fail clearly and early on bad input<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<\/p>\n<p>On the backend and ClickHouse:<\/p>\n<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>What we measured<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>How<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Why it mattered<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Query efficiency<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>ClickHouse rows read per row written, and per page returned, from system.query_log<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Exposes queries whose cost grows with the experiment instead of the request<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>CPU<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Backend pod core-seconds per upload; ClickHouse CPU per query type<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Shows what each upload costs the server, not just how long it takes<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Memory<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>ClickHouse memory per query; memory-limit errors<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Decides whether concurrent reads succeed or fail with HTTP 500<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Latency<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Backend request p50; ClickHouse query duration<\/p><p><\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Separates server time from client time<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Concurrency<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Throughput and backend CPU as client threads increase<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Shows whether extra parallelism buys throughput or just contention<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<\/p>\n<p>We tracked those KPIs because client timings only show how long the client waited, not what the server spent in the meantime or where that time went.<\/p>\n<h3>Test Scenarios<\/h3>\n<p>Once we knew what we were looking for, we prepared the test scenarios: large data-heavy ones, small ones, large numbers of items with small payload per item, and edge cases. We checked every run item by item against the server, so we didn&#8217;t add any run that lost or failed items to the comparison.<\/p>\n<\/p>\n<p>The next step was an environment where we could run these scenarios many times, with as few external factors as possible. To make every measurement reproducible, we ran the client and the server on dedicated infrastructure, with no other load and no other components.<\/p>\n<\/p>\n<p>Finally, timings say that something is slow, but they don\u2019t say where. Each layer had its own instrumentation, and we located every bottleneck with it rather than guessing.<\/p>\n<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Layer<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>What we used<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>What it located<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Backend<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>OpenTelemetry metrics: request latency, throughput, and resource use per endpoint<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Whether time went to the backend at all, and which endpoint paid for it<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>ClickHouse<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>system.query_log\u00a0grouped by query shape (rows read, duration, memory, CPU), EXPLAIN, and table-level metrics<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>The aggregates query reading 99.4% of all rows during a bulk upload; a paging scan reading 1.24M rows to return 2,000<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>SDK<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Profiling (cProfile), per-phase CPU timing, a memory sampler, and requests-in-flight counts<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>85% of read time spent converting responses into models; four upload passes taking 17.4 s before the first request<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<h2>What we found, and how we fixed it<\/h2>\n<p>Each finding follows the same path, in the order we worked through it:<\/p>\n<ul>\n<li>Symptom: A finding started from a number that looked wrong, like an upload slower than its data justified, a client idle while it should have been sending, memory growing with the dataset, or a server doing more work than the data required. The symptom said that something was off, not why.<\/li>\n<li>How we located it: We traced the symptom to its cause across layers, including client CPU, requests in flight, backend request times, the ClickHouse query log, and lock and thread metrics. Pinpointing the cause helped us understand the findings.<\/li>\n<li>The fix: The change and the release that shipped it.<\/li>\n<li>What it bought: We ran the same scenarios again, before and after, on the same environment, measuring both the client and the backend. Each finding&#8217;s tables come from their own interleaved runs so that the same release can differ by a few seconds from one finding to the next. Each fix also moved the bottleneck elsewhere, and the next finding starts there. <\/li>\n<\/ul>\n<p><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-thiago-3.png\"\/><\/p>\n<h3>Finding 1: Dataset versioning serialized every batch behind one lock<\/h3>\n<p>Symptom: On the 100K small items, the backend, not the client, set the pace: the client waited on every batch, and batches took longer the further the upload went. <\/p>\n<p>How we located it: Opik <a href=\"https:\/\/www.comet.com\/docs\/opik\/evaluation\/advanced\/manage_datasets#understanding-dataset-versioning\">datasets are versioned<\/a>. Every upload batch lands in a dataset version, and the backend determines which incoming items are new and which already exist so it can maintain the version\u2019s count. The whole batch write ran inside a per-dataset lock; the lock had been added to prevent concurrent uploads from racing on the dataset\u2019s latest version pointer and causing silent dropping of rows. But it covered far more than that race. A single dataset.insert()\u00a0of ~100K rows is one version creation followed by 99 appends, so 99 of 100 batches queued on a lock they did not need. And everything inside the lock added up, batch after batch:<\/p>\n<ul>\n<li>The new-versus-existing check, the aggregates query that read 99.4% of all rows during a bulk upload, read the whole version. To classify 1,000 incoming items, each batch read every item already in the version, so batch k read k \u00d7 1,000 rows and the cost grew quadratically over an upload.<\/li>\n<li>The version counters took three MySQL round trips per batch: read the version, write totals computed in Java, read it back.<\/li>\n<\/ul>\n<p>The fix, in two steps:<\/p>\n<ul>\n<li>Version 2.2.45: Signed atomic counter updates in one SQL statement, instead of three MySQL round trips<\/li>\n<li>Version 2.2.46:\u00a0Appends into an existing version run without the lock; only version creation keeps it, re-checked inside<\/li>\n<\/ul>\n<p>What it bought:\u00a0With the 2.2.46 lock change, the same SDK uploading the same 100K small items into a new dataset went from 89.1 s to 41 s, a 2.17\u00d7 improvement. These gains are for many small items, where the server was the bottleneck. With 30K large, deeply nested items, the same releases all took ~1030 s. Before 2.2.46, the lock was taken once per batch (832 times), but it cost only about 3 s in total, because the client kept only ~0.2 requests in flight. On that workload, the client was the bottleneck, which led us to Finding 2.<\/p>\n<h3>Finding 2: With the lock gone, the bottleneck moved to the client<\/h3>\n<p>Symptom: Once the server stopped serializing batches, it could accept uploads faster than the SDK could produce them. The bottleneck was no longer in the backend or ClickHouse; it was in the client.<\/p>\n<p>How we located it: The client-side instruments showed two different bottlenecks, one for each kind of data. On small items, the client mostly waited; it used 20 s of CPU across a 1 min 45 s upload and averaged 0.81 requests in flight, meaning it sent one batch and waited for the response, then sent the next. On large items, the client was busy instead, using about 36 minutes of CPU across a 39-minute upload to prepare each item before sending it.<\/p>\n<p>The fix, in two steps:<\/p>\n<ul>\n<li>Version 2.2.48: Send batches concurrently; uploads went from one thread to four by default, so up to four batches are in flight at once and the waits between them overlap. Users control it with the <a href=\"https:\/\/www.comet.com\/docs\/opik\/evaluation\/advanced\/manage_datasets#tuning-sdk-throughput\">num_threads<\/a>\u00a0argument in Dataset.insert(). The SDK only does this against backends that include the Finding 1 fix, where concurrent batches of the same upload are safe.<\/li>\n<li>Version 2.2.48:\u00a0Make deduplication optional; before sending anything, the SDK hashes every item to drop duplicates. When adding to a dataset that already exists, it first downloads and hashes every existing item. A periodic job adding to the same large dataset pays that on every run. Using deduplication=False\u00a0skips both steps when the caller knows the items are unique.<\/li>\n<\/ul>\n<p>What it bought:\u00a0Uploads finished 2.58\u00d7 faster on small items and 2.30\u00d7 on large ones, with the same client CPU. The client did the same work, but leveraging the available resources, on the same backend, uploading into a new dataset:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Dataset<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>SDK (default threads)<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Upload time<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Requests in flight<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Client CPU<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>ClickHouse inserts running at once<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.47 (1 thread)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 45 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.81<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>20.0 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.36<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.48 (4 threads)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>40.7 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.85<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>20.1 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1.2\u20131.7<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.47 (1 thread)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>39 minutes<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.09<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>36 minutes<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.03<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.48 (4 threads)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>17 minutes<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.21<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>36 minutes<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.06\u20130.07<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>Turning off deduplication saved 35 s of client CPU on large items, but made no measurable difference on the small one. On large items, four threads could not help with the preparation itself: the client still spent about 8 min 45 s preparing items before its first request. That is where Finding 3 starts.<\/p>\n<h3>Finding 3: Stream the upload instead of loading it<\/h3>\n<p>Symptom: Even with four threads sending, large uploads still took half their time before the first request. On 30K large items, about 8 min 45 s of a 17-minute upload passed before anything was sent, and the client held 17.8 GiB of memory while it waited.<\/p>\n<p>How we located it: Two instruments pointed at the same place; the client memory grew with the source size, meaning the SDK held every item at once. Also, the delay before the first request, on the client, matched the delay before the first insert in ClickHouse\u2019s query log. Nothing reached the backend until the SDK had read the whole source and split it into batches. The time was spent loading, not sending.<\/p>\n<p>The fix, in two steps:<\/p>\n<ul>\n<li>Version 2.2.61: Stream the source. Items are now read, batched, and sent as they go: each batch leaves as soon as it is full, and no items are kept once their request has been sent. Users get this by passing a generator, or any other iterator that produces items on demand, to Dataset.insert(), or by loading the file with <a href=\"https:\/\/www.comet.com\/docs\/opik\/evaluation\/advanced\/manage_datasets#inserting-items-from-a-jsonl-file\">Dataset.read_jsonl_from_file()<\/a>. A list still works, but it is already fully in memory before the upload starts, so it gains nothing.<\/li>\n<li>Version 2.2.61:\u00a0Keep the validation guarantee, and make it optional. These methods promised that every item was checked before any request was sent to the backend, and streaming could not quietly drop that. So by default, read_jsonl_from_file()\u00a0now makes one fast validation pass over the file and then streams it, the same guarantee, at a fraction of the memory. Also, with\u00a0validate_before_upload=False, the SDK starts streaming immediately and stops at the first bad item. That suits files that were already validated when they were generated, and because datasets are versioned, you fix a bad upload by uploading a new version.<\/li>\n<\/ul>\n<p>What it bought: In the 30K large items case, uploading the file went from running out of memory to about 3 minutes with the default settings, and about 2 min 40 s without the validation pass. On small items, streaming cut client memory by about 3\u00d7 and client CPU by about 4\u00d7; on large items, memory fell from running out to under 200 MiB. On the 100K small items case, uploaded from a file, on the same backend:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>SDK<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Upload time<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Client CPU<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Peak client memory<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Requests in flight<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.48 (loads the whole file)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>38.5 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>19.7 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>384 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.88<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.61, validation on (default)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>32.7 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4.6 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>127 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3.80<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.61, validation off<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>31.8 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3.3 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>127 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3.94<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<\/p>\n<p>The backend saw the difference too; its first insert arrived 8 s into the upload with 2.2.48, 3 s with 2.2.61, and 2 s when the validation pass was off. On small items, the pass cost about 1.2 s of the client CPU and had no measurable effect on upload time.<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>30K large items dataset<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Upload time<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Client CPU<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Peak client memory<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>First request after<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.48<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>OOM<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p><\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>all of the machine\u2019s available memory<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>never<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.61, validation on (default)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3 min 12 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3 min 12 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>193 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>32\u201343 s<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.61, validation off<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 min 39 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 min 41 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>193 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 s<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>Uploading the same 30K large items from a file was worse than the in-memory upload in Finding 2. In 2.2.48, read_jsonl_from_file()\u00a0parsed every line, turned the whole list back into one JSON string, and parsed it again, doing several full copies of the whole file that sat in memory at once. Also, in that case, the validation pass was no longer free. It cost about 33 s, which is exactly what validate_before_upload=False\u00a0gives back. Then, with the client much lighter and almost all four requests in flight on small items, the next question was whether more of them would help. That led us to Finding 4.<\/p>\n<h3>Finding 4: More concurrency, now that the producer can keep up<\/h3>\n<p>Symptom:\u00a0After streaming, the client almost always had a request waiting to go out. On the 100K small items, it kept about 3.8 of its 4 threads busy, so the thread count became the limit, and the backend seemed far from busy.<\/p>\n<p>How we located it:\u00a0Two measurements had to agree before adding threads was worth trying. The SDK has two parts on the upload path: the producer, a single thread that reads, validates, deduplicates, and fills batches, and the dispatcher, a pool of threads that sends them. The producer had capacity to spare. On the small items, it used around 17% of one core. The requests in flight were set at the thread limit, not below it. Together, they meant the client could prepare batches faster than four threads could send them.<\/p>\n<p>The fix: Raise the default number of threads in SDK 2.2.62. Dataset uploads went from 4 to 8 threads, and bulk experiments did too. Users control it with the same num_threads\u00a0argument as before, and a lower value caps the requests in flight.<\/p>\n<p>What it bought. It depends on who was waiting for whom. On the same backend, uploading from a file into a new dataset:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Data<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Threads<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Upload time<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Requests in flight<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>ClickHouse insert concurrency<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Producer thread, share of one core<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4 (2.2.61)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>32.4 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3.80<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~1.9<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>15\u201319%<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>8 (2.2.62)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>27.6 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>7.41<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~3.9<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>17\u201321%<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4 (2.2.61)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3 min 10 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1.56<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.8<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>99\u2013100%<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>8 (2.2.62)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3 min 11 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1.55<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.8<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>99\u2013100%<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>On small items, the producer had capacity to spare, so it used the extra threads; almost eight requests were in flight, the backend\u2019s inserts overlapped twice as much, and the upload finished 1.18\u00d7 faster than before. On the other hand, with large items, nothing changed; the producer thread ran at 100% on one core for the whole upload and couldn&#8217;t fill even two requests at a time. The producer itself was now the bottleneck; that is where Finding 5 starts.<\/p>\n<h3>Finding 5: Move compression off the producer<\/h3>\n<p>Symptom: On large items, more dispatcher threads changed nothing: the producer ran at 100% of one core for the whole upload, and requests in flight stayed around 1.5.<\/p>\n<p>How we located it:\u00a0Splitting client CPU by threads showed where the time went. On the 30K large items, the producer used 3 min 11 s of CPU in a 3 min 11 s upload, while all the dispatcher threads together used 2 s. The producer was also gzip-compressing every batch as it built it, and compression is the one that doesn\u2019t need to run in item order. It could run anywhere.<\/p>\n<p>The fix:\u00a0In SDK 2.2.63, compression moved to the dispatcher; each dispatcher thread now compresses the batch it is about to send, so batches are compressed concurrently, leaving the producer to read and prepare them.<\/p>\n<p>What it bought:\u00a0On the 30K large items, uploads finished 1.31\u00d7 faster. About 47 s of compression left the producer and spread across the dispatcher threads, so more requests were in flight, and the backend\u2019s inserts overlapped twice as much. Uploading datasets from file into the system:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Data<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>SDK<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Upload time<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Producer CPU<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Dispatcher CPU<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Requests in flight<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Peak client memory<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.62<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3 min 11 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>3 min 11 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1.57<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>193 MiB<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.63<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 min 25 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 min 24 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~46 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.8<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~350 MiB<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.62<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>29.1 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>5.2 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.2 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>7.4<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>130 MiB<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.63<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>26.8 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>5.3 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.8 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>7.1<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>138 MiB<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>The trade-off was memory, since raw batches now wait for a worker to compress them, so peak client memory nearly doubled during a large-item upload. On small items, compression had never been significant, and the difference stayed within run-to-run variation. On the other hand, on large items, the producer was still busy 100% of the time with what it had left, like reading, validating, deduplicating, and serializing. This is the starting point for Finding 6.<\/p>\n<h3>Finding 6: A faster serializer<\/h3>\n<p>Symptom:\u00a0After moving compression, the producer on large items was still at 99% of one core, and each row still cost it about 4.7 ms. The only way forward was to make that work cheaper.<\/p>\n<p>How we found it:\u00a0We measured the producer\u2019s CPU time per large item at about 4.7 ms. Most of it was serialization, which ran twice per item: once for the request body and once for the context hash that deduplication computes. Unlike compression, serialization couldn\u2019t move to the dispatcher threads. gzip releases Python\u2019s global interpreter lock while it works, but the JSON encoder holds it so that more threads wouldn\u2019t have made it faster.<\/p>\n<p>The fix, in two steps:<\/p>\n<ul>\n<li>Version 2.2.65: Serialize with orjson\u00a0where it is available. Python\u2019s standard json library is not the fastest encoder, so the SDK now uses orjson\u00a0for both request bodies and the deduplication hash. There is nothing to configure; 2.2.65 installs orjson\u00a0by default wherever a prebuilt package exists, and anywhere it is not available, the SDK falls back to the standard library, set up to produce the same values earlier releases sent. <\/li>\n<\/ul>\n<ul>\n<li>Version 2.2.65:\u00a0Write bytes directly; each serialization used to produce text and then convert it to bytes, so orjson now writes bytes directly and removes that conversion. Part of the gain comes from the faster library, and part from removing the intermediate conversion.<\/li>\n<\/ul>\n<p>What it bought: In the 30K large items case, the producer\u2019s work per row fell from 4.7 ms to 2.6 ms, and uploads finished 1.57\u00d7 faster. Uploading from a file into a new dataset with the default settings:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Data<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>SDK<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Upload time<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Producer CPU per row<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Producer busy<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Requests<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Requests in flight<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Peak client memory<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.63<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 min 25 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4.78 ms<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>99%<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>923<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.9<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~350 MiB<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.65<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 33 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.65 ms<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>86%<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>832<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4.7<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~615 MiB<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.63<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30.7 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.053 ms<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>18%<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>7.2<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>138 MiB<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.2.65<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>31.2 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.043 ms<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>14%<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>7.3<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>185 MiB<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>For the first time, the producer was not pinned at 100%, so the dispatcher kept more requests in flight and the backend\u2019s inserts overlapped more. The more compact output of orjson also packed the same items into fewer requests. The trade-off was memory again: peak client memory on large items rose from about 350 MiB to about 615 MiB, with more batches in flight at once. On small items, work per row fell too, but upload time didn\u2019t change because the backend already set the pace. This led us to Finding 7.<\/p>\n<h3>Finding 7: Stop binding parameters on the backend<\/h3>\n<p>Symptom:\u00a0With the client no longer the bottleneck, the backend set the pace on small items. Each 1K-row request spent 2 s in the backend, but only 1.01 s of that was related to ClickHouse.<\/p>\n<p>How we located it:\u00a0The gap between the backend\u2019s time per request and ClickHouse\u2019s insert latency was the hint. Much of the request passed before ClickHouse even saw the insert. A JFR profile of the backend during a bulk upload showed where that time went. The ClickHouse R2DBC driver\u2019s parameter binding resolves every bind(name, value)\u00a0with namedParameters.indexOf(name), a linear scan over all of the statement\u2019s parameter names. Bulk inserts render one placeholder per column per row, so that list grows with the batch, and binding a full batch becomes O(n\u00b2).<\/p>\n<p>The fix, in two steps:<\/p>\n<ul>\n<li>Versions 2.2.73\u20262.2.79: Stream rows through ClickHouse\u2019s v2 client, so the backend now sends rows to ClickHouse as JSONEachRow, so there are no parameter names to look up and no inlined statement to parse, all behind the ANALYTICS_DB_BULK_INSERT_V2_CLIENT_ENABLED\u00a0toggle, which is off by default.<\/li>\n<li>Version 2.2.101:\u00a0Removes the toggle, and the R2DBC bulk APIs now use the new client.<\/li>\n<\/ul>\n<p>What it bought:\u00a0Up to 3.7\u00d7 faster uploads, depending on how many rows each request carries. Uploading from a file into a new dataset with the default settings, using version 2.2.65 vs 2.2.86 with the v2 client toggle on:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Data<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Rows per request<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Backend insert path<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Upload time<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Backend time per request<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>ClickHouse time per insert<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Client CPU<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1K<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>R2DBC, bound parameters<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>28.9 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2.08 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1.01 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4.6 s<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1K<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>v2 client, JSONEachRow<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>7.7 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.36 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.32 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4.5 s<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~36<\/p><p><\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>R2DBC, bound parameters<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 29 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.47 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.23 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 min 9 s<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>~36<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>v2 client, JSONEachRow<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 20 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.30 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>0.13 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>2 min 6 s<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>The gain depends on the batch size because the cost of binding grows with the square of the rows in a request. So it lands hardest on full batches. Small items fill the SDK\u2019s 1K-item batch, and each request\u2019s backend time fell almost 6\u00d7, from 2.08 s to 0.36 s. Large items reach the request size limit at around 36 rows, so binding was already cheap.<\/p>\n<p>On top of this fastest configuration, we turned off the SDK\u2019s validation pass and deduplication, which is safe when the file is already known to be valid and duplicate-free. The large-item upload fell from 1 min 20 s to 46 s (1.74\u00d7 faster), and the small-item upload fell from 7.7 s to 6.6 s (1.17\u00d7 faster).<\/p>\n<h3>Finding 8: Experiments shared the bottlenecks, so they shared the fixes<\/h3>\n<p>Symptom:\u00a0Experiments were as slow as datasets on the first test runs using SDK 2.2.48; uploading 100K small items into an experiment took 5 min 34 s. With 30K large items, the upload stored only about 37% of the items after 30 minutes in all three runs.<\/p>\n<p>How we located it:\u00a0We used the same instruments pointed at the same places. On upload, the client was busy rather than waiting, so we checked, and SDK preparation was again the bottleneck; the backend wrote experiment items, traces, and spans through the same R2DBC binding as dataset items. Reads were also expensive; pages read almost the entire experiment to be built.<\/p>\n<p>The fix, in six steps:<\/p>\n<ul>\n<li>Version 2.2.62:\u00a0Raise the SDK default threads for the experiment dispatcher<\/li>\n<li>Version 2.2.70: Replace the four full passes the SDK made before sending anything (17.4 s of single-threaded CPU on a ~120K-item upload) with a single streaming pass, and add the validate_before_upload=False\u00a0flag.<\/li>\n<li>Version 2.2.78:\u00a0Increase the number of items fetched per page, allow concurrent page fetching using the num_threads\u00a0argument on Experiment.get_items(), and stop parsing each page into generated models, which had taken ~85% of the client&#8217;s read time.<\/li>\n<li>Version 2.2.79:\u00a0Ship the v2 client for all entities involved in the experiment ingestion.<\/li>\n<li>Version 2.2.82:\u00a0Make each page cost its size, not the whole experiment. Before, a paging scan read 1.24M rows to return the page size.<\/li>\n<li>Version 2.2.143:\u00a0Build each page on light columns first, and join the heavy input, output, and metadata once per page, so a page of large items no longer carries every payload through the grouping.<\/li>\n<\/ul>\n<p>What it bought:\u00a0Uploading and reading back an experiment became 19.6\u00d7 faster. We compared the initial version against the final one (SDK 2.2.86 on backend 2.2.86 with v2 client toggle on), uploading 100K small items into a new experiment linked to a new dataset and then reading them back, each with the SDK default settings:<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Step<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>First version<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Final version<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Faster<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Upload<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>5 min 34 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>18.4 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>18.2\u00d7<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Read back<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4 min 27 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>12.3 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>21.7\u00d7<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Upload and read<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>10 min 2 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30.7 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>19.6\u00d7<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>The upload used a tenth of the client CPU (2 minutes down to 12.3 s), and peak memory fell from 1.2 GiB to 264 MiB. The read now takes 53 requests instead of 1029. With each page reading only its own rows, concurrent reads of the 100K small item experiment no longer failed with HTTP 500. For the 30K large items, we lowered num_threads\u00a0from the default eight to four, since each page of large items uses much more memory. The reading now takes about 5 minutes. For the 30K large items, the gap is wider: the first version stored only 37% of the items after 30 minutes, while the final one uploaded all 30K in around 2 minutes.<\/p>\n<h2>Conclusion<\/h2>\n<p>We set out to make periodic evaluation on large datasets practical in Opik. At the start, uploading a large dataset took about 40 minutes for 30K rows; uploading from a large file couldn&#8217;t run on small machines. An experiment took minutes to write, minutes to read back.<\/p>\n<p>No single change got us there; each improvement removed a bottleneck and exposed the next one:<\/p>\n<ul>\n<li>The backend lock that serialized every batch<\/li>\n<li>A client that sent one batch at a time<\/li>\n<li>An SDK that loaded the whole source before sending anything<\/li>\n<li>Lack of concurrency to use resources more efficiently once the producer could keep up<\/li>\n<li>Compression, then serialization, on the producer thread<\/li>\n<li>Parameter binding in the backend<\/li>\n<\/ul>\n<p>The improvements spanned the SDK, the backend, and ClickHouse. What made this work was a methodical process of measuring, assessing, evaluating, and applying changes on both sides of the wire every time. So we always knew whether the client or the backend was setting the pace. The ~30\u00d7 was a compounding effect; it was never the goal, but the outcome of an iterative loop process. <\/p>\n<p>We uploaded 30K large items into a new dataset with the SDK\u2019s default settings. With version 2.2.46, it took 39 minutes and 17.4 GiB of client memory. With version 2.2.86, with the v2 toggle on, it took 1 min 20 s and 438 MiB: 29.5\u00d7 faster, with 40\u00d7 less memory. <\/p>\n<p><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-thiago-1.png\"\/><\/p>\n<p>Every other path got at least 13\u00d7 faster. Each figure is the median of three or four interleaved runs on the benchmark environment, with every upload checked item by item against what the backend stored.<\/p>\n\n\n<figure class=\"wp-block-table\" style=\"overflow-x:auto;margin:28px 0;\"><table class=\"has-fixed-layout\" style=\"border-collapse:collapse;width:100%;font-size:14px;line-height:1.5;\"><thead><tr><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Path<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Workload<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Before<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>After<\/p><\/th><th colspan=\"1\" rowspan=\"1\" scope=\"col\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;background:#edf3f4;font-weight:700;color:#142e34;\"><p>Change<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Dataset upload<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>39 minutes, 17.4 GiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 20 s, 438 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>29.5\u00d7 faster, 40\u00d7 less memory<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Dataset upload<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items, from a 4.8 GB file<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>could not run (out of memory)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 20 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>now possible<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Dataset upload<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 45 s, 398 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>7.7 s, 187 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>13.6\u00d7 faster<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Experiment upload<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>5 min 34 s, 1.2 GiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>18.4 s, 264 MiB<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>18.2\u00d7 faster<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Experiment read<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>100K small items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>4 min 27 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>12.3 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>21.7\u00d7 faster<\/p><\/td><\/tr><tr style=\"background:#f8fafb;\"><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Experiment upload<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><p><\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>37% stored after 30 minutes<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>1 min 57 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>over 15\u00d7 faster<\/p><\/td><\/tr><tr><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>Experiment read<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>30K large items<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>not possible (upload never completed)<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>5 min 1 s<\/p><\/td><td colspan=\"1\" rowspan=\"1\" style=\"border:1px solid #d7dfe3;padding:12px 14px;text-align:left;vertical-align:top;white-space:normal;\"><p>now possible<\/p><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p>Both columns upload the same items read from the same file. Before: SDK 2.2.47 on backend 2.2.46, where Dataset.insert()\u00a0collected every item into memory before sending the first one. With that SDK, the file loader ran out of memory on the large file. After: SDK 2.2.65 on backend 2.2.86 with the v2 client, uploading with Dataset.read_jsonl_from_file(), which streams. The 30K large-item read ran on backend 2.2.143, with four concurrent pages.<\/p>\n<p>With validation and deduplication also turned off, the large dataset upload took 46.0 s and the small one 6.6 s.<\/p>\n<p>On 100K small items, a full evaluation cycle (uploading the dataset, writing the experiment, and reading it back) went from 12 minutes to under 40 seconds. On 30K large items, the same cycle previously couldn&#8217;t be completed, and now finishes in about 8 minutes. That is the kind of short cycle that was our goal.<\/p>\n<p><img decoding=\"async\" alt=\"\" src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-thiago-2.png\"\/><\/p>\n<h2>What&#8217;s next<\/h2>\n<p>We&#8217;ll keep improving the ingestion and retrieval pipelines so that any kind of evaluation and testing can run after every change to an LLM or agent application monitored by Opik. We also recommend checking our blog on <a href=\"https:\/\/www.comet.com\/site\/blog\/ai-evaluation\/\">AI Evaluation<\/a>. <\/p>\n<\/p>\n<p>We also welcome you to contribute to Opik on our <a href=\"https:\/\/github.com\/comet-ml\/opik\">GitHub repository<\/a>, with code or by opening an issue when you find something that can be improved.<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<div>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Opik is Comet\u2019s open-source platform for tracing, evaluating, testing, and monitoring LLM and agent applications. It tracks every call an application makes, including prompts, tool calls, retrievals, and LLM outputs. This gives teams tools to understand the application\u2019s behavior, find where it goes wrong, and identify what to improve, from the developer workstation to production. [&hellip;]<\/p>\n","protected":false},"author":155,"featured_media":20935,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[9],"tags":[],"coauthors":[379],"class_list":["post-20929","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How we made Opik dataset and experiment ingestion 30\u00d7 faster - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How we made Opik dataset and experiment ingestion 30\u00d7 faster\" \/>\n<meta property=\"og:description\" content=\"Opik is Comet\u2019s open-source platform for tracing, evaluating, testing, and monitoring LLM and agent applications. It tracks every call an application makes, including prompts, tool calls, retrievals, and LLM outputs. This gives teams tools to understand the application\u2019s behavior, find where it goes wrong, and identify what to improve, from the developer workstation to production. [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-17T00:14:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-11-17T00:15:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-30x-faster-featured.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"720\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Thiago Hora\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Thiago Hora\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"25 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How we made Opik dataset and experiment ingestion 30\u00d7 faster - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/","og_locale":"en_US","og_type":"article","og_title":"How we made Opik dataset and experiment ingestion 30\u00d7 faster","og_description":"Opik is Comet\u2019s open-source platform for tracing, evaluating, testing, and monitoring LLM and agent applications. It tracks every call an application makes, including prompts, tool calls, retrievals, and LLM outputs. This gives teams tools to understand the application\u2019s behavior, find where it goes wrong, and identify what to improve, from the developer workstation to production. [&hellip;]","og_url":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2026-11-17T00:14:43+00:00","article_modified_time":"2026-11-17T00:15:48+00:00","og_image":[{"width":1280,"height":720,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-30x-faster-featured.jpg","type":"image\/jpeg"}],"author":"Thiago Hora","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Thiago Hora","Est. reading time":"25 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/"},"author":{"name":"Caleb's agent Loggins","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/299089a7fc32f84ad7a936215255514f"},"headline":"How we made Opik dataset and experiment ingestion 30\u00d7 faster","datePublished":"2026-11-17T00:14:43+00:00","dateModified":"2026-11-17T00:15:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/"},"wordCount":5145,"commentCount":0,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-30x-faster-featured.jpg","articleSection":["Product"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/","url":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/","name":"How we made Opik dataset and experiment ingestion 30\u00d7 faster - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-30x-faster-featured.jpg","datePublished":"2026-11-17T00:14:43+00:00","dateModified":"2026-11-17T00:15:48+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-30x-faster-featured.jpg","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-30x-faster-featured.jpg","width":1280,"height":720,"caption":"Opik ingestion up to 30\u00d7 faster: parallel data batches stream into a database."},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/improve-opik-ingestion-30x\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"How we made Opik dataset and experiment ingestion 30\u00d7 faster"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/299089a7fc32f84ad7a936215255514f","name":"Caleb's agent Loggins","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4b4b0a956e2c1c2858a4fd825f80b6e5","url":"https:\/\/secure.gravatar.com\/avatar\/782b3102d18eed861e22e63b2e70cae25d48619dfc6a1e667914d21455f59bba?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/782b3102d18eed861e22e63b2e70cae25d48619dfc6a1e667914d21455f59bba?s=96&d=mm&r=g","caption":"Caleb's agent Loggins"},"sameAs":["http:\/\/comet.com"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/caleb-agent\/"}]}},"jetpack_featured_media_url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2026\/11\/opik-ingestion-30x-faster-featured.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/20929","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/155"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=20929"}],"version-history":[{"count":2,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/20929\/revisions"}],"predecessor-version":[{"id":20937,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/20929\/revisions\/20937"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/20935"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=20929"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=20929"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=20929"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=20929"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}