{"id":2398,"date":"2021-07-07T09:51:00","date_gmt":"2021-07-07T17:51:00","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/"},"modified":"2021-07-07T09:51:00","modified_gmt":"2021-07-07T17:51:00","slug":"comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/","title":{"rendered":"Issue 8: GitHub’s Copilot, Distill’s Hiatus, Parameter Counts in ML Research"},"content":{"rendered":"\n

GitHub’s new “AI Pair Programmer”, Distill Pub announces a hiatus, examining parameter counts in ML research, and why robots aren’t a solution to the search and rescue mission in Miami’s tragic condo collapse.<\/em><\/h3>\n\n\n\n

Welcome to issue #8 of The Comet Newsletter!\u00a0<\/p>\n\n\n\n

This week, we share details and our perspective on GitHub\u2019s new \u201cAI Pair programmer<\/a> and Distill Pub\u2019s announced hiatus<\/a>.\u00a0<\/p>\n\n\n\n

Additionally, we highlight an interesting analysis of parameter counts in machine learning research<\/a>, as well as a reflection on why robots aren\u2019t cut out for helping with the unfolding crisis at the site of the recent Miami condo collapse<\/a>.<\/p>\n\n\n\n

Like what you\u2019re reading? Subscribe here.<\/a><\/h3>\n\n\n\n

And be sure to follow us on Twitter<\/a> and LinkedIn<\/a> \u2014 drop us a note if you have something we should cover in an upcoming issue!<\/p>\n\n\n\n

Happy Reading,<\/p>\n\n\n\n

Austin<\/p>\n\n\n\n

Head of Community, Comet<\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY<\/strong> | WHAT WE’RE READING | PROJECTS | OPINION<\/p>\n\n\n\n

GitHub introduces \u201cCopilot: your AI pair programmer\u201d<\/a><\/h2>\n\n\n\n

 <\/p>\n\n\n\n

\n
\"\"\n
Image Source<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n

GitHub announced a new project, Copilot, that\u2019s been built on top of OpenAI\u2019s Codex. The new AI system uses context prompts provided by the developer, in the form of comments, docstrings, function names, or even code to synthesize or complete the code for a given functionality.<\/p>\n\n\n\n

Copilot can generate code from comments, autofill repetitive code, and even write tests. Copilot also comes with a dedicated VSCode package and works inside GitHub\u2019s codespaces.\u00a0<\/p>\n\n\n\n

We see Copilot as the convergence of Microsoft\u2019s investments in OpenAI and GitHub,\u00a0 leveraging the advanced language modeling capabilities of GPT-3 and the vast amount of code available on GitHub as training data.\u00a0<\/p>\n\n\n\n

GitHub has also included a detailed report<\/a> on whether Copilot simply recites code from the training set or truly generates something novel\u2014a dynamic which has generated a bit of controversy in the community (read more about that here<\/a> and here<\/a>).\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n\n\n\n

The service is still a work in progress, and Github is currently offering access via technical preview to developers in order to test and improve the system.<\/p>\n\n\n\n

Read the full announcement here<\/a>.<\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY | WHAT WE’RE READING<\/strong> | PROJECTS | OPINION<\/p>\n\n\n\n

Distill, an interactive online journal for ML publications, announces indefinite hiatus<\/a><\/h2>\n\n\n\n

 <\/p>\n\n\n\n

\n
\"\"\n
The Distill Logo<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n

Distill<\/a>, the seminal interactive online journal for Machine Learning publications, will be taking a year-long hiatus that could extend indefinitely.\u00a0<\/p>\n\n\n\n

For the past 5 years, Distill has been a venue for authors to explore methods for scientific communication\u2014from Gabriel Goh\u2019s interactive exposition of momentum<\/a>, to feature visualizations that attempt to interpret the information represented in the neurons of a network<\/a>.\u00a0<\/p>\n\n\n\n

The editorial team at Distill cite changes in the way they think about the function of Distill\u2014as well as burnout from the volunteer team\u2014as the reason for this hiatus. Specifically, they talk about the inherent conflict between being a highly-editorialized journal while also trying to achieve their other goals of mentorship and providing a neutral venue for publication.\u00a0<\/p>\n\n\n\n

They believe that Distill isn\u2019t living up to their expected standards for author experience. Burnout has made the review process much slower and similar to a typical journal. The team said it was becoming unclear to them if their review process added enough value to justify the additional time cost to authors.\u00a0\u00a0<\/p>\n\n\n\n

The team also feels like self-publication (for example on ArXiv) might be a large fraction of the future of scientific communication. There are tremendous benefits to journal-led review, but the team asks whether \u201ctraditional journal-led peer review is the most effective way to achieve these benefits? And is it worth the enormous costs it imposes on editors, reviewers, authors, and readers?\u201d<\/p>\n\n\n\n

Extending on the notion of self-publishing being a big part of the future, the team noted that \u201cSelf-publication can move very fast. It doesn\u2019t require a paper to fit into the scope of an existing journal. It allows for more innovation in the format of the paper, such as using interactive diagrams as Distill does. And it aligns incentives better.\u201d<\/p>\n\n\n\n

Distill\u2019s approach to scientific communication has inspired a wave of open source and commercial projects aimed at better communication for Machine Learning. Tools like Google\u2019s Model Cards<\/a>, Streamlit\u2019s UI\u2019s for Data Science<\/a>, and even Comet\u2019s own Reports<\/a> all leverage interactive visualizations to communicate ideas in machine learning.\u00a0<\/p>\n\n\n\n

Despite the hiatus, the Distill template is still open source and freely available, and the Editorial Team is hopeful that others will run with it and attempt to build on their work.\u00a0 \u00a0

Read the complete comments from the Distill team here<\/a>.<\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY | WHAT WE’RE READING | PROJECTS <\/strong> | OPINION<\/p>\n\n\n\n

Examining parameter counts in machine learning<\/a><\/h2>\n\n\n\n

 <\/p>\n\n\n\n

\n
\"\"\n
Image Source<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n

How do you measure the collective progress of AI technology? Currently, no one has great answers for this question, but researchers at the Alignment Forum have suggested using the parameter count of models as a \u201cproxy metric that allows us to understand one way in which models have become more complex over time, and ground predictions of how the field will progress in the future.\u201d\u00a0<\/p>\n\n\n\n

In particular, they hope to tease apart how much of the advancement is coming from algorithmic improvements, and how much is coming from increases in model complexity.\u00a0\u00a0\u00a0<\/p>\n\n\n\n

The trend they found was that parameter counts for models have been increasing over time. The two main insights from this work are<\/p>\n\n\n\n

    \n
  1. \u201cThere was no discontinuity in any domain in the trend of model size growth in 2011-2012. This suggests that the Deep Learning revolution was not due to an algorithmic improvement, but rather the point where the trend of improvement of Machine Learning methods caught up to the performance of other [non-]ML methods.\u201d<\/li>\n
  2. From 2016 onwards, there has been discontinuity in the scale of language models. \u201cAn eyeball estimate of the slope of progress suggests that the doubling rate was between 18 and 24 months from 2000 to 2016-2018 in all domains, and between 3 and 5 months from 2016-2018 onward in the language domain.\u201d\u00a0<\/li>\n<\/ol>\n\n\n\n

    The researchers conclude their post with a critique of their dataset and data collection methods, a list of open questions based on the uncovered trends, and links to the dataset and code used to produce the article.<\/p>\n\n\n\n

    Read the full report here.<\/a><\/p>\n\n\n

    ——————————–<\/center>\n\n\n

    INDUSTRY | WHAT WE’RE READING | PROJECTS | OPINION<\/strong><\/p>\n\n\n\n

    Why robots can\u2019t be counted on to find survivors in the Florida building collapse<\/a><\/h2>\n\n\n\n

    On June 24th, a portion of the 12-story Champlain Towers South condominium building in Surfside, Florida, suffered a catastrophic partial collapse<\/a>. In this article, Evan Ackerman explains the reason why robots have seen limited use in a rescue situation such as this.\u00a0<\/p>\n\n\n\n

    The article covers the complexities of collapsed structures and the safety considerations of trying to send a robot into such environments.\u00a0<\/p>\n\n\n\n

    Dr. Robin Murphy, the director of the Humanitarian Robotics and AI Laboratory at Texas A&M, says that \u201cusing robots to explore rubble of collapsed buildings is, for the moment, not possible in any kind of way that could be realistically used on a disaster site.\u201d\u00a0<\/p>\n\n\n\n

    \u201cRubble is a wildly unstructured and unpredictable environment,\u201d Dr Murphy continued. \u201cMost robots are simply too big to fit through rubble, and the environment isn\u2019t friendly to very small robots either, since there\u2019s frequently water from ruptured plumbing making everything muddy and slippery, among many other physical hazards.\u201d<\/p>\n\n\n\n

    Disaster scenarios in robotics research also assume that objectives are achievable if the right path is followed. Real disasters are not like that. The act of moving rubble out of the way can cause the robot to dislodge structures that dramatically change the navigation path\u2014or worse, cause injury. Robotic sensing systems are also not well designed for extremely close quarters, and visual sensors such as cameras can become rapidly damaged by dirt and debris.\u00a0<\/p>\n\n\n\n

    Rescue robots would need to be able to understand the internal structure of the rubble\u2014and the consequences of altering the structure\u2014in order to be effective. On top of these technical challenges, Dr. Murphy also notes that the ethical expectation is for robotics technology used in these situations to be fully mature.\u00a0<\/p>\n\n\n\n

    \u201cIt\u2019s obviously unethical to take a research-grade robot into a situation like the Florida building collapse and spend time and resources trying to prove that it works.\u201d\u00a0<\/p>\n\n\n\n

    Read the full story here.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

    A new Comet integration with TensorBoardX, Tesla at CVPR, a new e-book on ML job interviews, more intelligent multi-task learning, and more<\/p>\n","protected":false},"author":1,"featured_media":2304,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[10],"tags":[],"coauthors":[130],"class_list":["post-2398","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-industry"],"yoast_head":"\nComet Github Copilot Distill Hiatus ML Parameters ML Parameters<\/title>\n<meta name=\"description\" content=\"Comet Newsletter 8: GitHub\u2019s new \u201cAI Pair Programmer\u201d, Distill Pub announces a hiatus, examining parameter counts in ML research, and more.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Issue 8: GitHub's Copilot, Distill's Hiatus, Parameter Counts in ML Research\" \/>\n<meta property=\"og:description\" content=\"Comet Newsletter 8: GitHub\u2019s new \u201cAI Pair Programmer\u201d, Distill Pub announces a hiatus, examining parameter counts in ML research, and more.\" \/>\n<meta property=\"og:url\" content=\"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2021-07-07T17:51:00+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Comet-Newsletter-Ad-1200x627-02.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"627\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Austin Kodra\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Austin Kodra\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Comet Github Copilot Distill Hiatus ML Parameters ML Parameters","description":"Comet Newsletter 8: GitHub\u2019s new \u201cAI Pair Programmer\u201d, Distill Pub announces a hiatus, examining parameter counts in ML research, and more.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/","og_locale":"en_US","og_type":"article","og_title":"Issue 8: GitHub's Copilot, Distill's Hiatus, Parameter Counts in ML Research","og_description":"Comet Newsletter 8: GitHub\u2019s new \u201cAI Pair Programmer\u201d, Distill Pub announces a hiatus, examining parameter counts in ML research, and more.","og_url":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2021-07-07T17:51:00+00:00","og_image":[{"width":1200,"height":627,"url":"http:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Comet-Newsletter-Ad-1200x627-02.jpg","type":"image\/jpeg"}],"author":"Austin Kodra","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Austin Kodra","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/#article","isPartOf":{"@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/"},"author":{"name":"engineering@atre.net","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/550ac35e8e821db8064c5bd1f0a04e6b"},"headline":"Issue 8: GitHub’s Copilot, Distill’s Hiatus, Parameter Counts in ML Research","datePublished":"2021-07-07T17:51:00+00:00","mainEntityOfPage":{"@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/"},"wordCount":1384,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Comet-Newsletter-Ad-1200x627-02.jpg","articleSection":["Industry"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/","url":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/","name":"Comet Github Copilot Distill Hiatus ML Parameters ML Parameters","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/#primaryimage"},"image":{"@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Comet-Newsletter-Ad-1200x627-02.jpg","datePublished":"2021-07-07T17:51:00+00:00","description":"Comet Newsletter 8: GitHub\u2019s new \u201cAI Pair Programmer\u201d, Distill Pub announces a hiatus, examining parameter counts in ML research, and more.","breadcrumb":{"@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Comet-Newsletter-Ad-1200x627-02.jpg","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Comet-Newsletter-Ad-1200x627-02.jpg","width":1200,"height":627,"caption":"Comet Newsletter"},{"@type":"BreadcrumbList","@id":"http:\/\/www.comet.com\/site\/blog\/comet-newsletter-8-github-copilot-distill-hiatus-ml-parameters\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Issue 8: GitHub’s Copilot, Distill’s Hiatus, Parameter Counts in ML Research"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/550ac35e8e821db8064c5bd1f0a04e6b","name":"engineering@atre.net","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/027c18177377edf459980f0cfb83706c","url":"https:\/\/secure.gravatar.com\/avatar\/d002a459a297e0d1779329318029aee19868c312b3e1f3c9ec9b3e3add2740de?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d002a459a297e0d1779329318029aee19868c312b3e1f3c9ec9b3e3add2740de?s=96&d=mm&r=g","caption":"engineering@atre.net"},"sameAs":["https:\/\/live-cometml.pantheonsite.io"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/engineeringatre-net\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/2398","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=2398"}],"version-history":[{"count":0,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/2398\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/2304"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=2398"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=2398"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=2398"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=2398"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}