Guides
Masked image generation
֍

Learn how to use masked image generation techniques using StableDiffusionXLControlNet and StableDiffusionXLInpaint.

image
Generate illusions
no background
Edge variation and inpainting

We'll cover two workflows that use masked image generation:

  1. Generate illusions: StableDiffusionXLControlNet with the illusion method generates illusions incorporating the mask.
  2. Edge variation and inpainting: StableDiffusionXLControlNet with the edge method generates variations of an image using edge detection, and StableDiffusionXLInpaint generates inside the mask.

First, initialize Substrate:

Python
TypeScript

from substrate import (
Substrate,
RemoveBackground,
StableDiffusionXLControlNet,
StableDiffusionXLInpaint,
)
s = Substrate(api_key=YOUR_API_KEY)

Here's the original image:

original

1. Generate illusions

This workflow uses RemoveBackground to generate a mask, followed by StableDiffusionXLControlNet with the illusion method to generate a two images incorporating the mask into a view of the ocean from above:

Python
TypeScript

mask = RemoveBackground(image_uri="https://media.substrate.run/logo-sq.png", return_mask=True)
controlnet = StableDiffusionXLControlNet(
image_uri=mask.future.image_uri,
control_method="illusion",
conditioning_scale=1,
prompt="sunlit bright birds-eye view of the ocean, turbulent choppy waves",
num_images=2,
)
res = s.run(controlnet)

image
no background

Experimenting with different prompts can produce striking results:

image
no background

2. Edge variation and inpainting

This workflow uses StableDiffusionXLControlNet to generate a variation of the original, and StableDiffusionXLInpaint to generate a variation inpainting inside a mask.

Python
TypeScript

original = "https://media.substrate.run/logo-nopad-bg-white.png"
controlnet = StableDiffusionXLControlNet(
image_uri=original,
control_method="depth",
prompt="silver disco ball, 4k",
conditioning_scale=0.8,
num_images=1,
)
inpaint = StableDiffusionXLInpaint(
image_uri=controlnet.future.outputs[0].image_uri,
mask_image_uri=mask.future.image_uri,
prompt="towers in the futuristic ancient solarpunk city of atlantis",
num_images=1,
)
res = s.run(inpaint)

image
res.get(controlnet)
no background
res.get(inpaint)