{"id":8086,"date":"2023-11-02T10:08:41","date_gmt":"2023-11-02T18:08:41","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=8086"},"modified":"2025-04-24T17:04:47","modified_gmt":"2025-04-24T17:04:47","slug":"dataset-preparation-meets-experiment-and-model-management-with-superb-ai-and-comet","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/dataset-preparation-meets-experiment-and-model-management-with-superb-ai-and-comet\/","title":{"rendered":"Dataset Preparation Meets Experiment and Model Management with Superb AI and Comet"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

When it comes to machine learning projects, the hard truth is that training just one model on one version of a dataset won\u2019t result in a production-ready model. The entire ML lifecycle is, by its nature, deeply iterative and interdependent<\/strong>. For a given project, dataset creation and model development will undoubtedly require numerous cycles.<\/p>\n

And what\u2019s more, making changes to one part of your ML workflow changes every part of your ML workflow<\/a>.<\/p>\n

New training data? You will need to run a few more model training experiments to understand how this new data will affect model performance. Your model isn\u2019t performing well? You might have to return and collect ground truth data samples, adjust your labels, or make other dataset changes. This feedback loop<\/strong> between dataset collection\/management and model training\/experimentation is one of the most important \u2014 and potentially costly \u2014 parts of the ML lifecycle.<\/p>\n

To help you build better models faster, you and your team will need tools and capabilities that allow you to make intelligent adjustments each step of the way<\/strong> \u2014 all while having high visibility into your workflows, as well as the ability to collaborate and reproduce your work at every step.<\/p>\n

This is the power that tools like Superb AI<\/a> and Comet<\/a> offer when put to work in unison. In this article, we\u2019re going to take a look at how these two tools can work together to help you speed up and improve two different but deeply connected stages of the ML lifecycle: (1) dataset collection and preparation + (2) model training and experimentation.<\/p>\n

Additionally, we\u2019ll show how you and your team can use Superb AI and Comet to create a feedback loop between model predictions, dataset iterations, and model retraining processes.<\/p>\n

Superb AI Dataset Preparation Platform<\/h1>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Source<\/em><\/a><\/figcaption><\/figure>\n

Superb AI has introduced a revolutionary way for ML teams to drastically decrease the time it takes to deliver high-quality training datasets for computer vision use csases. Instead of relying on human labelers for a majority of the data preparation workflow, teams can now implement a much more time- and cost-efficient pipeline with the Superb AI Suite<\/a>.<\/p>\n

A typical data preparation pipeline might contain the following steps:<\/p>\n

    \n
  1. Data ingestion: <\/strong>Input data (images and videos) are often extracted from various sources. Users can upload input data into Superb Suite as raw files, via the cloud (AWS S3 or GCP), or via Superb Suite\u2019s SDK\/API.<\/li>\n
  2. Ground-truth data creation: <\/strong>Having a small amount of initial ground-truth data (data with correct labels) is crucial to kickstart the labeling process. Users can create these ground-truth samples using Superb\u2019s built-in simple annotation tool with filtering capability. Superb Suite supports classification, detection, and segmentation tasks for bounding boxes, polylines, polygons, and key points for both images and videos.<\/li>\n
  3. Automatic labeling: <\/strong>Superb AI\u2019s customizable auto-label technology<\/a> uses a unique mixture of transfer learning, few-shot learning, and self-supervised learning \u2014 allowing the model to quickly achieve high levels of efficiency with small customer-proprietary datasets. And because the custom auto-label has broad applications, it can be used to swiftly jump-start any project, whether that be labeling your initial dataset for training or labeling your edge cases for retraining. This will drastically reduce the time it takes to prepare and deliver datasets.<\/li>\n
  4. Labeled data delivery: <\/strong>The review and audit process of data labels is vital for the overall quality of the dataset. In reality, it is almost impossible to review every label manually. Superb AI Suite streamlines the review process by taking advantage of the label accuracy measures estimated by multiple machine learning models. After passing through this rigorous quality control process, the final labeled data is delivered to the MLOps pipeline.<\/li>\n<\/ol>\n

    Comet\u2019s Experiment Management Platform<\/h1>\n

    Machine learning addresses problems that cannot be well specified programmatically. Traditional software engineering allows strong abstraction boundaries between different components of a system in order to isolate the effects of changes.<\/p>\n

    Machine learning systems, on the other hand, are entangled with a host of upstream dependencies, such as the size of the dataset, the distribution of features within the dataset, data scaling and splitting techniques, the type of optimizer being used, etc.<\/p>\n

    Because ML systems lack a clear specification, data collection is an imperfect science, and effective machine learning models can be incredibly complex, experimentation is necessary<\/strong>.<\/p>\n

    The goal of the experimentation process is to understand how incremental changes affect the system. Rapid experimentation over different model types, data transformations, feature engineering choices, and optimization methods allows us to discern what is and isn\u2019t working.<\/p>\n

    Because Machine learning is an experimental and iterative science<\/a>, diligent tracking of these multiple sources of variability<\/a> is necessary. Manually tracking these processes can be quite tedious and is further exacerbated when the size of an ML team grows and collaboration between members becomes a factor. It is well known that reproducibility<\/a> is an issue in many machine learning papers, and while steps are being taken to address these issues<\/a>, as humans, we are often prone to oversight.<\/p>\n

    This is where a platform like Comet comes into the picture. Comet is an Experiment Management Platform<\/a> that helps practitioners automatically track, compare, visualize and share their experiments, source code, datasets, and models.<\/p>\n

    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/figure>\n

    A typical workflow for experimentation with Comet contains the following steps:<\/p>\n

      \n
    1. Define the project scope and relevant metrics<\/strong>: This is usually the most challenging part of the model development process. It involves talking to various stakeholders and clearly establishing the outcomes expected from the model and how these will be measured.<\/li>\n
    2. Log the relevant dataset to Comet as an Artifact<\/strong>: Comet Artifacts<\/a> is a tool used to track the lineage of datasets, as well as other data assets produced during the course of experimentation (model checkpoints, intermediate datasets, etc.).<\/li>\n
    3. Experiment and iterate over different model types, data transformations, feature engineering choices, and optimization methods<\/strong>: We\u2019ve already mentioned that in order to discern the type of model that would work best for your dataset, experimentation is necessary. Automated tracking of these variables allows teams to iterate on their models rapidly.<\/li>\n
    4. Evaluate the model in production<\/strong>: Once a candidate model is chosen for deployment, it should be deployed to a production environment and monitored in order to assess its performance on real-world data and identify gaps in the model\u2019s performance.<\/li>\n
    5. Update the data<\/strong>: The performance of a model will inevitably degrade over time as the input data changes. In order to address these performance gaps, it\u2019s necessary to update training datasets for this model with new data.<\/li>\n
    6. Retrain the model<\/strong>: Once the training datasets are updated, repeat steps 1\u20134 to update the model.<\/li>\n<\/ol>\n

      Building a Data-to-Model Pipeline and Feedback Loop with Superb AI and Comet ML<\/h1>\n
      \n
      \n
      \"\"<\/figure>
      <\/picture><\/div>\n<\/div>\n<\/figure>\n

      Coupled together, Superb AI and Comet cover data preparation and model experimentation workflows, respectively. As observed in the workflow diagram above:<\/p>\n

        \n
      1. Given the raw data from your data sources, you can use the Superb AI platform to ingest the data, create a small ground-truth dataset, label that dataset with the auto-label technology, manually audit the hard labels, and generate a labeled training dataset.<\/li>\n
      2. Then, the labeled dataset flows from the Superb AI platform to the Comet platform.<\/li>\n
      3. You now can use the Comet platform to create Artifacts of the labeled dataset, run model experiments and evaluate model performance with the Artifacts, visualize model predictions, and surface failure prediction cases.<\/li>\n
      4. Next, you can feed the failure prediction cases from Comet to Superb AI for ground-truth labeling and kickstart the model retraining procedure.<\/li>\n<\/ol>\n

        You can keep workflows between DataOps and MLOps teams separate \u2014 while enabling cross-team collaboration by preserving the visibility and auditability of the entire data-to-model pipeline across teams. With this pipeline in place, teams can increase the velocity and opportunity for seamless collaboration between scientists and engineers for machine learning workflows.<\/p>\n

        Conclusion<\/h1>\n

        Intelligent platform choices make machine learning development much more feasible \u2014 especially as you\u2019re scaling your ML strategy. With Superb AI\u2019s data preparation capabilities and Comet\u2019s model development capabilities, your ML teams can:<\/p>\n