Getting Started

Performance Improvements

  • Dataset and experiment transfers now default to 8 threads — The Python SDK’s bulk paths shipped with inconsistent worker counts: dataset reads (get_items(), stream_items()) and Dataset.insert() ran on 4 threads, while Experiment.batch_upload_items() uploaded batches sequentially unless you passed num_threads yourself. All three now default to 8, the setting we benchmark against — the experiment upload path benefits most, since it went from sequential to parallel. Every call still takes num_threads if you want to push harder or ease off; see Tuning SDK throughput. Note that raising it much further does not help: on a 119,903-item upload, 16 threads measured slightly slower than 8, because the SDK saturates a CPU core serializing and compressing payloads before thread count becomes the limit. Parallel dataset upload still requires an Opik backend of 2.2.8 or newer, and falls back to a sequential upload against anything older.