{"id":7147,"date":"2023-08-18T11:57:56","date_gmt":"2023-08-18T19:57:56","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7147"},"modified":"2025-04-29T15:20:13","modified_gmt":"2025-04-29T15:20:13","slug":"image-inpainting-for-sdxl-1-0-base-refiner","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/image-inpainting-for-sdxl-1-0-base-refiner\/","title":{"rendered":"Image Inpainting for SDXL 1.0 Base Model + Refiner"},"content":{"rendered":"\n

In this article, we\u2019ll compare the results of SDXL 1.0 with its predecessor, Stable Diffusion 2.0. We\u2019ll also take a look at the role of the refiner model in the new SDXL ensemble-of-experts pipeline and compare outputs using dilated and un-dilated segmentation masks. Finally, we\u2019ll use Comet to organize all of our data and metrics. Feel free to follow along in the <\/span>full-code tutorial here<\/a><\/strong>, or, if you can\u2019t wait to see the final product, check out <\/span>the public project here<\/a><\/strong>.<\/span><\/p>\n\n\n\n

 <\/p>\n\n\n\n

\"Realistic
Image created by author with SDXL 1.0 base + refiner and edited in Canva; seed = -732, prompt = “astronaut in outerspace, photorealistic.”<\/figcaption><\/figure>\n\n\n\n
\n
Follow along with the Colab!<\/a><\/div>\n\n\n\n
Create a free Comet account!<\/a><\/div>\n<\/div>\n\n\n\n

What is SDXL 1.0 and why should I care?<\/h2>\n\n\n\n

SDXL 1.0 is the new foundational model from Stability AI that\u2019s making waves as a <\/span>drastically-improved<\/span><\/a> version of Stable Diffusion, a latent diffusion model (LDM) for text-to-image synthesis. As the newest evolution of Stable Diffusion, it\u2019s blowing its <\/span>predecessors<\/span><\/a> out of the water and producing images that are competitive with black-box SOTA image generators like Midjourney.<\/span><\/p>\n\n\n\n

\"A
The results are in and practitioners prefer the SDXL 1.0 pipeline by a wide margin!<\/figcaption><\/figure>\n\n\n\n

The improvements are the result of a series of intentional design choices, including a 3x larger UNet-backbone, more powerful pre-trained text encoders, and the introduction of a separate, diffusion-based refinement model. The refiner model improves the visual fidelity of samples using a post hoc image-to-image diffusion technique first proposed in <\/span>SDEdit<\/span><\/a>. In this tutorial, we\u2019ll use SDXL with and without this refinement model to get a better understanding of its role in the pipeline. We\u2019ll also compare these results with outputs from Stable Diffusion 2.0 to get a broader picture of the improvements introduced in SDXL.<\/span><\/p>\n\n\n\n

\"Graphic
The SDXL 1.0 mixture-of-experts pipeline includes both a base model and a refinement model.<\/figcaption><\/figure>\n\n\n\n

But these improvements do come at a cost; SDXL 1.0 involves an impressive 3.5B parameter base model and a 6.6B parameter refiner model, making it one of the largest open image generators today. This increase is mainly due to more attention blocks and a larger cross-attention context, since SDXL uses a second text encoder (OpenCLIP ViT-bigG<\/a> with CLIP ViT-L<\/a>). <\/span><\/p>\n\n\n\n

SDXL 1.0 and the future<\/h3>\n\n\n\n

Still, the announcement is an exciting one! In the last year alone, Stable Diffusion has served as a foundational model in fields spanning 3D classification<\/a>, controllable image editing<\/a>, image personalization<\/a>, synthetic data augmentation<\/a>, graphical user interface prototyping<\/a>, reconstructing images from fMRI brain scans<\/a>, and music generation<\/a>. SDXL 1.0 promises to continue Stable Diffusion\u2019s tradition of widening the realm of generative AI possibilities.<\/span><\/p>\n\n\n\n

\"In
In one incredible application of a previous version of Stable Diffusion<\/a>, researchers showed volunteers the top row of images above. They then fed fMRI scans of their brain activity to the model, which successfully created the bottom row of reconstruction images. What new advancements will SDXL bring?<\/figcaption><\/figure>\n\n\n\n

SDXL 1.0 vs. the world<\/h3>\n\n\n\n

So, how does SDXL 1.0 fare in the wider world of text-to-image generative AI tools? According to SDXL, very well. In fact, it\u2019s now considered the world\u2019s best open image generation model. And while Midjourney still seems to have an edge as the crowd favorite, SDXL is certainly giving it a run for its money as a free open source alternative.<\/span><\/p>\n\n\n\n

SDXL 1.0 is both open-source and open-access, meaning it\u2019s free to use, as long as you have the computational resources to do so. But it doesn\u2019t require much; all of the images in this article were generated using Google Colab\u2019s A100 GPU. And according to Stability AI, SDXL 1.0 will even work effectively on consumer GPUs with just 8GB of VRAM, making generative text-to-image models more accessible than ever before.<\/span><\/p>\n\n\n\n

But what makes SDXL image outputs better than ever? According to Stability AI, SDXL offers:<\/span><\/p>\n\n\n\n