February 25, 2022
Notes from the eight session of a brand new Office Hours series: Seven Simple Steps to Standardizing the Experiment with guests Dr. Doug Blank, Jacques Verre, Dhruv Nair and Michael Cullan.
Welcome to another recap of the Comet ML Office Hours, powered by The Artists of Data Science! This week we’re covering Session 6 of our new series. This session took place Feb. 9th, 2022 and we were joined by Dr. Doug Blank from Comet and Tiffany Fabianac of AstraZeneca.
As a reminder, we’d love to see any and all of you at these fifty minute sessions—so feel free to register for upcoming Office Hours sessions here! As always, there’s a lot more in the full session (which you can find on our YouTube channel), so be sure to check it out, alongside clips from roundtables, webinars, and previous Office Hours.
This week we focused on how experiment management helps you iterate, build, and deploy models faster. The key takeaway from this week’s panel was reproducibility. Knowing the exact hyperparameters, tracking each change, and generally building reproducibility into your standard practice is vital.
When asked about building reproducibility, both panelists suggested melding it into the culture of the company or workflow. Doug suggested focusing on protecting against irreproducibility as true reproducibility is a myth. Hear more in the clips below:
Tiffany also suggested that rather than focusing on just being able to copy an experiment, practitioners should focus on maintaining their understanding of the model and the problems that it solves. As data will change and models will adjust, keeping the end-goal in mind can be a means of creating an iterative machine learning lifecycle.
As an example of how impossible it is to prepare for all outcomes, Doug told a story from his time teaching developmental robotics. The robot, which didn’t have memory, was supposed to travel down a hall and turn based on the color of the light.
What happened was a memory-less robot teaching itself on the job! Hear the full story in the clip below:
During the session, Doug mentioned the book The Extended Mind by Annie Murphy Paul.
As always, there’s more to be discussed and discovered. Check out Comet’s contributor-led publication Heartbeat as well as our YouTube, Twitter, and LinkedIn for more great information.
We run these virtual Office Hours every Wednesday at 11am EST (New York, NY). Completely free to attend and participate, and we’d love to see any and all of you there! We’ve got a great series planned and welcome questions for Harpreet or any of our guests via email to firstname.lastname@example.org.
But most importantly: