{"id":2513,"date":"2021-09-09T15:33:31","date_gmt":"2021-09-09T23:33:31","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/blog\/the-comet-newsletter-issue-17-graph-neural-nets-concept-drift-deep-dive-rethinking-large-language-models-and-more\/"},"modified":"2021-09-09T15:33:31","modified_gmt":"2021-09-09T23:33:31","slug":"the-comet-newsletter-issue-17-graph-neural-nets-concept-drift-deep-dive-rethinking-large-language-models-and-more","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/the-comet-newsletter-issue-17-graph-neural-nets-concept-drift-deep-dive-rethinking-large-language-models-and-more\/","title":{"rendered":"Issue 17: Graph Neural Nets, Concept Drift Deep-dive, Rethinking Large Language Models"},"content":{"rendered":"\n

Welcome to issue #17 of The Comet Newsletter!

In this week’s issue, we share a two-part deep dive on graph neural networks (part 1<\/a>; part 2<\/a>), as well as a re-evaluation of the idea that bigger = more complex when it comes to language models<\/a>.

Additionally, we explore an introduction to ML optimizers and compilers<\/a> and an in-depth guide to the ins and outs of concept drift once a model is deployed<\/a>.<\/p>\n\n\n\n

Like what you\u2019re reading? Subscribe here.<\/a><\/h3>\n\n\n\n

And be sure to follow us on Twitter<\/a> and LinkedIn<\/a> \u2014 drop us a note if you have something we should cover in an upcoming issue!<\/p>\n\n\n\n

Happy Reading,<\/p>\n\n\n\n

Austin<\/p>\n\n\n\n

Head of Community, Comet<\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY<\/strong> | WHAT WE’RE READING | PROJECTS<\/p>\n\n\n\n

Large language models aren’t always more complex<\/a><\/h2>\n\n\n\n

Whenever a new state-of-the-art language model arrives (think GPT-3), it’s generally accompanied by media coverage and analysis touting the number of parameters it was trained on (these days, in the 150+ billion range), as well as its complexity.<\/p>\n\n\n\n

But as Kyle Wiggers writes for VentureBeat, industry experts are becoming increasingly skeptical that the size of the models and the datasets they’re trained on correspond directly to improved performance.<\/p>\n\n\n\n

In terms of parameter counts and how the models are trained, this boils down to explicit “instruction tuning”, or the specific kinds of NLP tasks the model is trained to solve (i.e. translation, sentiment analysis). Interestingly, models trained in this fashion are still often able to generalize to other tasks, much like GPT-3. Wiggers cites the FLAN architecture (“Fine-tuned language network”) as an example of this dynamic.<\/p>\n\n\n\n

When it comes to dataset problems, Wiggers notes that “over-filtering” for data that achieves high classifier scores can actually lead to worse model performance. This kind of excessive optimization can lead to a “misalignment between proxy and true objective(s)” in what the model is designed to achieve, according to Leo Gao, data scientist at EleutherAI.<\/p>\n\n\n\n

Read the full story in VentureBeat<\/a>\u00a0<\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY | WHAT WE’RE READING<\/strong> | PROJECTS<\/p>\n\n\n\n

Inferring Concept Drift Without Labeled Data<\/a><\/h2>\n\n\n\n

In this incredibly thorough guide from\u00a0Andrew Reed<\/a>\u00a0and\u00a0Nisha Muktewar<\/a>\u00a0of Fast Forward Labs, the authors offer a deeply researched survey of the background, problematic nature, and approaches to solving what’s known as “concept drift” in production ML models. Put simply, concept drift refers to the fact that data that models are trained on often don’t reflect dynamic (and shifting) real-world conditions\u2014and therefore, resulting models, left alone, won’t be able to adapt and generalize to these changing conditions.<\/p>\n\n\n\n

This impressive guide hammers home the point that an ML team’s work isn’t done once a model has been deployed\u2014in fact, in most cases, the work is just beginning. And not only does this guide do a great job of clearly defining concept drift, it also provides practical approaches to solving the problem, a sample use case, and a number of additional concerns that ML teams should consider, ranging from ethical considerations, to handling concept drift within “big data” systems.<\/p>\n\n\n\n

Read the full report from Fast Forward Labs<\/a><\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY | WHAT WE’RE READING | PROJECTS<\/strong><\/p>\n\n\n\n

From Distill Pub: Graph Neural Networks Deep Dive<\/a><\/h2>\n\n\n\n

While we recently reported on Distill’s hiatus (which they are still on, apparently), we were treated to a bit of a surprise last week when they shared a 2-part series on graph neural networks, authored by several researchers at Google.<\/p>\n\n\n\n

Part 1 provides an overview of graph neural networks, including how they’re structured, how they work, and the kinds of problems they’re fit to solve.<\/p>\n\n\n\n

Part 2 zooms in on convolutions on graphs, which make up and impact the “building blocks and design choices of graph neural networks.”<\/p>\n\n\n\n