I have a ComfyUI workflow for Stable Diffusion 3.5 image generation and upscale. The…
- User
I have a ComfyUI workflow for Stable Diffusion 3.5 image generation and upscale. The initial image generated is 1280 x 720 pixels (let's call it initial prompt, initial image). Because SD can't directly generate images more than around 1 megapixel, I have to use some tricks. It also have an upscaler which contains : the upscale model itself - I use "4x Real SSDIR DAT GAN" (4xReal_SSDIR_DAT_GAN) - it shows better results than the popular "nmkd superscale", it's set to upscale the image by 2x, so the image is upscaled to 2560 x 1440, then it splits into 4 images grid (with 64 pixel overlap, it contains the part itself AND 64 pixels of the next image, so the edges WILL be consistent and won't be even noticed when reconstructing the image back), after that each image (1280 x 720) goes into LLM model recognition (currently I use Meta Llama 3.1 8b) with simple prompt "describe the image in formal tone within 100 words" - because the image is splitted it no longer contains ALL of the elements described in the generation prompt, so the new prompt is needed (new prompt 1 - 4 , new image 1 - 4). After that each of the splitted images with a text prompt describing it goes into Stable Diffusion module that is doing the image generation, with a denoise factor around 0.2 - so it will create a "new" image based on new prompts and images. Then all 4 generated images are split back into one 2560 x 1440. I've tried the "faster" ways without LLM, but I was not satisfied with the quality at all. I have 128 Gb of RAM and RTX 4080 GPU - the hardware won't be upgraded in any time soon. Generation time: around 45 seconds for SD image (initial one and for each of the 4 splitted), around 25 seconds for first upscale (SSDIR DAT GAN model), 90 seconds for the LLM work. So, that's around 6 minutes for 2560 x 1440 image. It can be scaled up (3 x 3 grid, 3840 x 2160 output, 9 images in split with 45 sec generation time + LLM work, or 4 x 4 grid - 5120 x 2880 , 16 images in split etc). Now the question to you: the LLM upscaler works fine, but I wonder if there are some quality improvements (speed is not necessary).
- Assistant
Your current workflow for generating and upscaling images using Stable Diffusion 3.5 and ComfyUI is quite comprehensive, leveraging large language models (LLMs) to maintain detail and consistency across high-resolution outputs. While this method produces satisfactory results, there are alternative techniques that might enhance image quality without necessarily increasing processing time.
1. High-Resolution Fix (HiRes Fix): HiRes Fix is an iterative refinement process that upscales images while preserving and enhancing details. It involves initial upscaling, inpainting to fill in missing details, denoising to reduce artifacts, and multiple iterations to refine the image progressively. This method can improve image quality without the need for LLMs, potentially reducing processing time[1].
2. ControlNet Tile Model: The ControlNet Tile model divides the original image into overlapping segments, upscales each segment individually, and then reconstructs the high-resolution image. This approach ensures localized enhancement and smooth transitions between segments, mitigating grid-like artifacts often seen in upscaled images. Integrating this model into your workflow could enhance image quality and consistency[2].
3. ResAdapter: ResAdapter is a domain-consistent adapter designed for diffusion models to generate images with unrestricted resolutions and aspect ratios. It enables the generation of high-resolution images without additional fine-tuning, preserving the original style and details. Implementing ResAdapter could streamline your workflow and improve output quality[3].
4. MegaFusion: MegaFusion extends existing diffusion-based text-to-image models towards efficient higher-resolution generation without additional fine-tuning. It employs a truncate and relay strategy to bridge denoising processes across different resolutions, allowing for high-resolution image generation in a coarse-to-fine manner. Integrating MegaFusion could enhance your workflow's efficiency and output quality[4].
5. Relay Diffusion: Relay Diffusion unifies the diffusion process across resolutions for image synthesis. It transfers a low-resolution image or noise into an equivalent high-resolution one via blurring diffusion and block noise, allowing the diffusion process to continue seamlessly in any new resolution without restarting from pure noise. Implementing Relay Diffusion could improve the coherence and quality of high-resolution images[5].
6. Mixed Precision Computation: Utilizing mixed precision computation can enhance performance by performing calculations in
float16precision while storing weights infloat32. This approach leverages faster operations on modern GPUs, potentially reducing processing time without compromising image quality[6].By exploring these alternatives, you may find methods that enhance image quality and efficiency in your Stable Diffusion workflow without relying on LLMs. Each technique offers unique advantages, and integrating them could lead to improved high-resolution image generation.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.