Beyond DALL-E: Advanced Image Generation Workflows with ComfyUI

Speaker 1 [00:07]

Yeah, thank you. Welcome. Yeah, in this talk, I'm going to show you what we can achieve if we lose ourself from the belief that image JNI just has to be a text-to-image interface and become more open for alternative input and generation methods. Yeah, I'm Rene. I'm from Innovex, an IT project center, and we do soft cloud infrastructure, software development in front and back end. We also do data and AI projects. We have six talks this year and over 20 attendees, so if you want to get in touch with us, just talk to us. Yeah, so back to topic. Overall, you can say text descriptions are very powerful. It's a quick and easy way to describe concepts of an image, and they do carry a lot of information. I mean, just the term summer or forest, if you think about it, it's a lot of information just in these two words. But if you care about precise coloring or image positioning or want to have custom shapes and objects, you're eventually going to hit a wall there. And what I'm going to show you is how we can use supplementary models and different generation techniques to generate images that, when we just use textual descriptions, are just impossible. So, textual descriptions are powerful but imprecise. That's what I want to show. And the tool I'm going to use here is CompuEye. It's open source, modular, image generation front end where you can create image pipeline, generation pipelines using a node graph-based interface. So you have nodes, and they have outputs and inputs, and you connect them using connection lines and build a pipeline this way. And because of that, creating plugins for it is very easy. So it's basically just a function. function, you have inputs, you have outputs, and you do something in between. And because of that, there's already a ton of plugins available, which is very nice. So if you want to use cutting edge technology or want to use features that are not implemented in there right now, there's a plugin available for sure. If you want to do image pre and post processing, you want to integrate new models or use ChatGPT to generate prompts for you, this This is everything what you can do with using plugins. And ConfUI itself can not only be used as a front end, you can also use it as a back end because it provides an API. So let's say you have built a pipeline and you want to use it productively or want to be triggered externally, you can use the API for that. So during this presentation, what I'm going to show you next, I'm running a ConfUI Instant which is hosted on RunPod IO. And if you want to replicate anything I've done here, I will provide a GitHub link at the end where you can find the workflows, the assets I've used, and the Docker image I've built. Yeah. But so let's get right into it, I would say. So we start with an anti-Canvas. This is ConfUI and where you can create the node stuff. And if we want to create a node, we double-click in the middle, and then we have a search window. So now we first want to create a Hello World example, which is, in this case, a simple text-to-image pipeline. The first thing, let's say we want to generate an image of a grumpy-looking cat sitting on a couch. So just something simple. We start off with creating nodes, and the first node we need is the model we want to use. So we double-click in the middle, enter load checkpoint, and there we have created a node to load the models. I'm going to use Stable Fusion, more precisely a fine-tuned version of it, which is called Juggernaut XL. I'm going to use Stable Diffusion XL here because it's a nice balance for generation speed and image quality, and I want to keep things fluid here. So Stable Diffusion provides, and this checkpoint here provides three models. The diffusion model itself, the Clip Text Encoder, and the Variational Autoencoder, which is converting the image from pixel space into latent space because Stable Diffusion is not working directly on pixel space but in latent representation of the image. And the next node we want is we want to enter a prompt. So we need to create a node to enter a prompt for it, which is called clipTextEncode. So this has one input, which is the clip model. We have it here. So we just drag and drop, and it's connected. The next thing we need is a node which actually generates our image, which is here called the case sampler. There is a more advanced one, but we leave it outside for now. We use the simple one. And the simple one needs four inputs. The first one is the positive conditioning, which is here. We provide it here. And make it green, just coloring, so we need to know what it is. And we can enter a prompt. Let's say a stunning photograph of a grumpy-looking cat sitting on a couch in a cozy living room. Some other descriptions, cinematic, vibrant colors, okay, like this. So the case sampler also needs a negative input. So I just copy it here, I color it red so we know it's a negative prompt, we don't need to do, which is, this is what we want to see in our image, this is what we don't want to see in our image. And we don't want to see, I don't know, text because it looks ugly. watermark, okay. We also connect the clip model to it, and now we connect it to the negative input. It also requires a model, which we got here from load checkpoint, so we connect it from load checkpoint to case sampler, and the last thing we need is the latent image. Because stable diffusion is actually an image-to-image model, so it starts with an image, denoises it, and does this in steps, and this is how it generates the image. So we need an image as a start. And we start off with an empty latent image here where it then uses this as a basis to generate the image. Stable effusion works with image size of 2024 times 2024, 2024, so like this, okay. And we connect it here to latent image. Okay. So we have the prompts. We have loaded the model. We do the generation. And we have an initial image. So now we want to actually see it. We can't use the latent image and we can't save it. So we need something to convert it back from latent image to pixel space. So we need the variational autoencoder for it and we need the decode node for it. So we connect it here, latent, and then it outputs a pixel image. It needs a model as an input, which is the variational autoencoder, so we connect it here. Okay. Last thing we need, we want to see the image. So we use a safe image node so we can actually see what we generated. Okay. And now we're good to go, basically. This is the very basic pipeline. We load the model, we enter the prompt, initial image, the sampler, decoder, and the preview. Now we can enter some settings here. This is the seed, basically the starting noise. I want to have it fixed. So if we make changes here, we can see what we actually did. And the next two settings I'm setting here is specific for the fine-tuned model I use. So if you look at the Hugging Face page for this model, you'll see that these are recommended settings. I'll just set it right here. Just take it as given here. OK, I'll increase the step size a bit. OK, now we can actually click on Run and generate the image. Here it does it in 30 steps this time, and there we go. Okay. We can look at here, and this is the image we generated. Okay. Just the cat looks a bit grumpy. This is what we want. Yeah. And now we can, like I said, this is the heavy world example. Now we can, if we want to do fancy stuff with it, we can extend this pipeline like we want. So, let's say we want to generate this cat in Ghibli style. So if you watched Reddit or something like this, it went viral when ChatGPT was able to generate these comic-style images. This was actually possible quite a while ago just using StableEffusion, but not stock StableEffusion with the base model and nothing else, but if you use LoRa. LoRa is a small supplementary model, which in this case is the size of 200 megabytes, and it injects features into the model if you want, so it modifies them slightly and makes it possible to generate things that the base model is not able to do. I'm going to use now preset examples just to speed things up, but actually it's very simple. Laura which is here load Laura and I've already selected the model and it takes the model and a clip model as an input and returns and modified version of it so I just connect the inputs here I got it okay and then I can here just set the updated once okay and now it's updated so this Laura here needs it needs a trigger word to actually to know that it's it should be in action I forget everything what they were precise wearing is the thing this is why I switched to another workflow which is basically the same we are still have everything like before we have added the Laura in between we have the checkpoint we have the positive and negative form that just call it a blue and we have the So, basically, if we look at here before and after, nothing really changed. Okay. We're going to use that. And this is the trigger prompt here. So, Laura is actually doing stuff. So, if I run generate, it's basically a similar prompt. I just removed some keywords which are contradicting with the comic style, style, like cinematic or photograph, which is not the case. Here we want to have a comic-style image, which is not a photograph, and this would contradict each other. And this is how simple you can use Ghibli-style images. You just load the LoRa, enter the trigger words, and this is the grumpy cat this time in Ghibli style. The next example I want to show you where I want to generate the Python logo as an island from top down. What would be the naive way? The naive way would, I'll switch to another example here, just the basic example without the lower again, set the prompt, a stunning photograph of a tropical island with the shape of the Python logo, a satellite photo, top down, aerial shot, coral reef, beach, forest, you name it. Let's see what happens if we try that out. Ah, beautiful Python logo. Yeah, it doesn't work. Why doesn't it work? Because Stable Diffusion doesn't know what the Python logo looks like, especially not if we want to have it as an island. So what can we do instead? What we can do is we have a starting image. We can, instead of generating everything from scratch, we can use the Python logo as a basis, change the colors, use this as a starting image, and just use a slightly less denoising amount to generate the image. Let me quickly switch to example. It might be a bit overwhelming at first, but it's not that bad. Let me quickly disable these. We just ignore it for now. We come to this later. So what do we have here? If you ignore this part here, it's basically the same as before. We have the load checkpoint, the text encoder, positive, negative prompt, the sampler, decoder, and the image. What changed is I reduced the denoise because we don't want StableFusion to change everything. But we have a very coarse starting image and then add the detail in it. So and we used an encoder for the variation autoencoder to encode our starting image from pixel space to latent space. So let me quickly start it. And we see so we load the Python logo here. I've done just some masking here and then I dyed the mask. the mask, and so we have a rough image what our island should look like. So here's the forest, here's the beach, and some water in between. This is what I want to have, but this looks not like an island, not really. So let's just do it for us. So this is basically just the initial image. I use this as a basis, and then some masking and coloring, and this is what happens here. So here in the prompt, I don't mention the Python logo at all because I don't need to. It doesn't know what the Python logo looks anyway. So we just say it's a tropical island and top down and everything. And what we have generated here looks a lot more already like the island we want to generate. But it's still not perfect. The edges are very rough and not precisely following what we want. But the coloring is right, at least. So to really get the edges correct, we can use something which is called a control net. A control net allows us to use input images to generate images that follow the input image. So we have specific control nets for different input images. For example, here I'm using a control net which uses Kenny images as the basis. So Kenny is just an edge detector. So if I run it here, it uses this starting image here and just detects the edges and returns an image which just contains the lines where the edges are. And the control net, which is here, just uses this as a basis and instructs Stable Diffusion to create an image that precisely follows these edges. And the next image we get here does look a lot more like the Python logo. So what we did was we used a starting image with a lower denoise value to get the coloring right, and then we used the control net to get the edges right. And the beauty of ConfUI is that you can combine things. Let's say you want to use an image with a LoRa and control net, and you want to use a starting image. Let's say, for example, here in the fourth example, I have a picture of myself in Japan, but I recognize that it does not look like Japan. I want to have it in Japan. So, what is the most obvious thing? I make it in Ghibli style. So, this is the pipeline, and it's basically nothing new for what I've shown here. We have the LoRa. We We have the basics part with positive coder, negative coder, case sampler, variational altering coder here, the decoder step. We have the control net, what we just used now, and we have a starting image. Now we just don't generate our image and do some masking and stuff, but it's basically the same. We just load the image. And if I run it here, so again, candy control net, we have the edges here, we have the starting image with a lower D noise. And this is Japan. Okay. Let me look at the time. Okay. So, the next example I want to show you is about fruits. So, let's say we have generated an image of a table with containing fruits, and I don't like grapes. I don't want the grapes in the image, and I want to get rid of them. So what can I do? What I can do is I can do inpainting. So I selectively just re-render a part of an image I want to change. So let's just catch up what's happening here. We have the clip text encoder here. We have the little checkpoint. We have the sampler, the decoder here. What's changed is that we have some nodes for inpainting. This is actually a plug-in which is very nice, which converts a regular stable diffusion model into an inpainting model. So you don't need dedicated inpainting diffusion models for it. And you can just convert stable diffusion models into inpainting models, which is very nice. And I used it here. But the most interesting part is actually here. I installed a plugin for Segment Anything, so basically what can I do if I want to remove the grapes? I can create a mask manually and then I do the inpainting, but I don't want to do that. I just want to have my grapes selected automatically by a prompt. And this is what Segment Anything can do in combination with Grounding Dino. This plugin, you can just enter a prompt, which is in this case grapes, and this plugin returns a mask for you, which just contains the grapes. Let me run this. I messed up. Okay. Now, it takes us some time to load the model. I can't explain what happens next in the meantime. So what happens now is, okay, it's quite fast. So here we have now selected only the grapes. Here this is our mask. I just do some erosion and some dilations to get rid of freckles and really catch the edges. And then I regenerate the image without the nasty grapes. And they're gone. Okay. But there's one thing I didn't mention. And this is what do you think what happens if I provide a grape-shaped mask? It will just generate the grapes in again. So what I did is I used the string where it said grapes, and I put it into the negative prompt also. So I say, okay, I don't want to see grapes in this image. And this is why it doesn't generate grapes in it. So let's say, okay, let's say you don't like grapes, but you don't like bananas for some reason. So you can just... Bananas. Okay. Should be good. Bananas. Okay. Yeah. Okay. Yeah, okay, now the grapes are in there again, and the image is regenerated, but this time without bananas. Come on. Yeah, there. Yeah, very nice. The last example I want to show you is you don't have to understand it fully, it's more Or like a showcase that you also can do low-level generation, and you can actually be very precise and very low-level operation in the generation you want to do. And for this, I'm going to show you an example here. There's a project which is called Illusion Diffusion, which is linked to my GitHub. Not by myself, but I would... yeah. Where someone created an image where on one side it looks like a sloth, and if you turn it upside down, it looks like a dog. And I like this idea. And I thought it was really cool, and I wanted to replicate it in CompuEye. And this is what I do, what I've done. But to really understand what's going on, I will quickly explain what's actually happening in this generation, because it's not that straightforward. So let's say we have two different generations. We start with the same noise, and then we do a sampler step, one with giraffe head and and one with Penguin. So they are completely independent from each other at first. So it does one step, and then we take the latent space, which returns out of it, and we combine them together. So because it's a flip image, we flip them upside down, we combine them, flip it back, and then feed it in into the next step. So with each step, we remove some noise out of the image and take control out of the diffusion model. And with each step, we combine the images together, and so they slowly converge into a single image, which represents both prompts, one from the one side and the other from the other side. And this is what I've done here. So in the next example, here, it's a bit bigger. But yeah, it's basically doing what I just told. So this node group here I've created is basically doing one step, then it returns a latent image, it rotates in one, then they combine, rotate back, and take this as a starting image for the next step. So if I run it, yeah. So slowly but steady, these two images will converge. It's a bit more complicated and fiddly to really get the right images, so I'm not doing one image, but I actually do 16 at the same time, and we'll see which image looks best. So and okay, first, and this is upside down. The same image is upside down. Okay, now let's pick a good one. I like this one. OK. So here you have a giraffe, and here you have a penguin. Yeah, and basically, this is what I wanted to show you. So what can we say? If you want to use this for simple tasks, like just text to image generations, it's too complicated. But if you want to use cutting edge models or things, or you want to have maximum controllability, you want to make use of the modularity and do low-level operations and automate, maybe, because you can use it as an API, this might be the right tool for you. And if you want to replicate anything, here's a QR code which you can scan through my GitHub link. You can try it. It should work. If someone gets a problem, I just also have this one. But yeah. Thank you.

Speaker 2 [24:19]

Thank you, Reni, for this wonderful talk. We have some questions on the slide, and I would want to address these. So the questions are, are you running these examples on your laptop? So if yes, what are the computation power required?

Speaker 1 [24:33]

Like I said, I'm running it on RunPodIO, which is like GCP, a cloud provider, and I'm just using SSH forwarding to forward the port into my local, and this is where I run it.

Speaker 2 [24:33]

Like I said, Okay, it's comfy UI open source. Hmm. Is it is it open source the comp I UI

Speaker 1 [24:51]

I understand what that's for. Open source. Yes, it is open source. Yes, sir.

Speaker 2 [24:56]

Have you used confi UI for a work project or just for fun?

Speaker 1 [25:01]

The first time I got in touch with it was for a project where we wanted to generate, to place products, images of products in a generated scenery. So let's say a box of cereals and we want to have it placed in a kitchen and we want during breakfast or something like this, this was a use case, or just creating stock photos for web pages or something like this.

Speaker 2 [25:28]

Okay, so one question is like this. It does not really look comfy Do the documents of comfy UI makes it any confier because comparing with stable diffusion web UI it is more complicated

Speaker 1 [25:41]

Yeah, there is a documentation. So you can learn it there. Also tons of YouTube videos. There's a steep landing curve, definitely. But if you get used to it, I love it. It's fun.

Speaker 2 [25:54]

Is it possible to export a workflow as a python script to add it into a django project or there are any other alternatives for using on my website? Great talk by the way.

Speaker 1 [26:03]

Thank you So if I said question correctly you want to use it as a back-end right to yeah like I said, this is possible you have an API and you can just host it somewhere and Basically use the different service to trigger the API to generate or to execute pre-generated Pre-created workflows

Speaker 2 [26:26]

Yeah, can config UI create videos?

Speaker 1 [26:29]

Yes, it's also possible, you can also create sounds, so for example Stable Fusion Audio is also working. There are tools which are easier to handle, especially for example for in-painting. Let's say for example you want to do in-painting, excessive in-painting and not just one, you want to generate a mask, but let's say you want to use it like Photoshop or something like this, there are better tools for it. But you can do it. And if you want to have specialized things, like I said, then this is the right tool.

Speaker 2 [26:59]

okay I think there are no more questions I'll address last one it's there is not there a loop option for flip diffusion

Speaker 1 [27:12]

Yeah, I guess you can create a plugin for it, like I said, very modular and you can just do it. But yeah, this workflow right here, yeah, if you want to have a loop, you need to use the case sampler, but because you need to flip it every time, it wasn't that easy, so I had to use this approach. It was more a showcase that you can do it.

Speaker 2 [27:37]

I think that's it. Let's give him a round of applause for this wonderful talk.