{"id":6366,"date":"2023-06-25T13:30:08","date_gmt":"2023-06-25T21:30:08","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=6366"},"modified":"2025-04-29T14:03:44","modified_gmt":"2025-04-29T14:03:44","slug":"sam-stable-diffusion-for-text-to-image-inpainting","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/sam-stable-diffusion-for-text-to-image-inpainting\/","title":{"rendered":"SAM + Stable Diffusion for Text-to-Image Inpainting"},"content":{"rendered":"\n

In this article, we\u2019ll leverage the power of SAM, the first foundational model for computer vision, along with Stable Diffusion, a popular generative AI tool, to create a text-to-image inpainting pipeline that we’ll track in Comet. Feel free to follow along with the full code tutorial in this Colab<\/a><\/strong> and get the Kaggle dataset here<\/a><\/strong>.<\/p>\n\n\n\n

Or, if you can\u2019t wait, check out the full public project here<\/strong><\/a>!<\/p>\n\n\n\n

\"Two
On the right, our original image of a frog. On the left, our output image where the frog has been replaced by a koala bear; image by author.<\/figcaption><\/figure>\n\n\n\n
\n
Follow along with the Colab!<\/a><\/div>\n\n\n\n
Create a free Comet account!<\/a><\/div>\n<\/div>\n\n\n\n

What is SAM?<\/h2>\n\n\n\n

Earlier this year, Meta AI caused another huge stir in the computer vision community with the release of their new open-source project: the Segment Anything Model (SAM). But, what makes SAM so special? <\/span><\/p>\n\n\n\n

SAM is a prompt-able segmentation system with results that are simply stunning. It excels in zero-shot generalization to unfamiliar objects and images without the need for additional training. It\u2019s also considered the first foundational model for computer vision, which is big news! We\u2019ll talk a little more about foundational models next. <\/span><\/p>\n\n\n\n

SAM was trained on a massive dataset of 11 million images with 1.1 billion segmentation masks, which Meta has also released publicly. But perhaps the best way to showcase SAM\u2019s groundbreaking capabilities is with a short demo:<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Meta
The Segment Anything Model (SAM) is famous for correctly identifying up to hundreds of individual masks per image. It can also generate multiple valid masks for ambiguous prompts; GIF from Meta AI.<\/span><\/figcaption><\/figure>\n\n\n\n

What are foundational models?<\/h2>\n\n\n\n

Foundation models are neural networks trained on massive unlabeled datasets to handle a wide variety of tasks. These powerful machine learning algorithms power many of the most popular Generative AI tools used today, including ChatGPT and BERT. <\/span><\/p>\n\n\n\n

Foundation models have made major strides in natural language processing, but until recently, haven\u2019t gained much traction in computer vision applications. That\u2019s because computer vision has struggled to find a task with semantically rich unsupervised pre-training, akin to predicting masked tokens for NLP. With SAM, Meta set out to change this.<\/span><\/p>\n\n\n\n

How to use SAM<\/h2>\n\n\n\n

The Segment Anything Model requires no additional training, so all we need to do is provide a prompt that tells the model what to segment in a given input image. SAM accepts a variety of input prompt types, but some of the most common ones include:<\/span><\/p>\n\n\n\n