Pix-Art Sigma + SDXL workflow (15 + 15 steps)

UPDATE : I included a special inpaint to add details and also a new method to re-imagine an input image (without control net or ip-adaptors). Also a sdxl only workflow and one with double control nets. Here i'm using the 8bit T5 from sd3. You can change that with a bigger model fp16 or full precision if you want.

This workflow does 15 steps (or 20) with pixart sigma and following with 15 steps (or 20) on sdxl. Sometimes i like to keep more of the base image and i run 10-12 steps in sdxl. Just play with the values.

Keep the loras of the sdxl model there, helps with anatomy. Playing with cfg for 4-8 for both models gives the best results.

Do not mistake this workflow with an upscaling one, this is not an upscale, this harmonizes both models by transferring latents and getting a coherent image without upscaling artefacts so you get the best from both worlds. You get excellent composition from Pix-art Sigma and then get to have excellent quality, colors, style, anatomy with the sdxl pass. Make sure to use a sdxl model with great anatomy and prompt comprehension. So you can use my sdxl model or bake DPO (direct preference optimization) into your favorite models. Not sure Pony counts as a model with good anatomy because it tries it's best to copy source art, and does not have good anatomy without doing so, but try and see, maybe i'm wrong.

Installation Instructions :

Install the latest torch and xformers in comfyui (recommended).

pip install torch torchvision torchaudio xformers --extra-index-url https://download.pytorch.org/whl/cu121

It is better to clone the comfyui repo in a folder, then install everything in a conda environment. That said if you don't want to use the comfyui portable edition and want to install xformers. This was the way i did it.

conda create --name comfyui
conda activate comfyui

Install extra models from comfyui manager (updating the node necessary)

Download the Sigma model from : Here

PixArt-Sigma-XL-2-1024-MS.pth 

Download the following T5 and rename them as seen : Here

Or if you wish to use the 8bit T5 download it from stability sd3 medium hf repo.

config.json 
model.safetensors.index.json 
model-00001-of-00002.safetensors 
model-00002-of-00002.safetensors 
t5xxl_fp8_e4m3fn.safetensors
UPDATE : you can use only the 8-bit T5 if you wish (4gb).

Download this VAE file from : Here

Rename it to pixart_sigma_vae.safetensors 
Update : The normal sdxl vae works too.

Place them inside the ComfyUI models directory as follows :

comfyui/models/
    /checkpoints
       > PixArt-Sigma-XL-2-1024-MS.pth
       > ReyMix_XL.safetensors  (or any other SDXL model)
    /t5
       > config.json
       > model.safetensors.index.json
       > model-00001-of-00002.safetensors
       > model-00002-of-00002.safetensors
    /clips
       > t5xxl_fp8_e4m3fn.safetensors
    /vae
       > pixart_sigma_vae.safetensors