This is a new VAE for SDXL which outputs images in the Rec. 2100 PQ color space in order to allow for HDR images. This should be used together with a model or LORA which takes advantage of it.
Important:
This will not create correct images out-of-the-box, as it will be interpreted as being in a wrong color space. You need to manually change the color space of the images after generation, otherwise images will look gray and boring.
Currently only Chromium based browsers support HDR and you will need to have a monitor that supports HDR and have it enabled in order to get any HDR effect.
HDR requires a higher bit depth and you can get posterization artifacts in 24-bit color. If you are making SDR images, you should use the normal VAE for better quality.
Rationale:
LORAs can be used to change the style of a model, but while it can approximate a new color space this way, it will not necessarily map every concept it knows correctly. A prompt specifying colors such as "red dress" can cause it to create the wrong color as it tries to use its old representation of it. This VAE is trained to keep the original latent representation as it has always been, and instead map the "red" color to the new color space in the VAE. This is way more robust than trying to handle all edge cases in the LORA training.
To support HDR, a LORA is trained just to push the generated latent outside the normal SDR range for concepts directly related to HDR.
Tools:
This VAE contains a new meta-data entry to specify its color space:
'modelspec.color_space': 'cicp:9,16,0,true',This is not yet standardized and image generators will incorrectly assume the image is in sRGB. To create correct looking images, you need to manually fix the images.
To override the color space for PNGs, I have created a python script you can run here:
https://github.com/spillerrec/rec2100-vae-training/blob/master/mark-png-as-hdr.py
This will produce grayish images on browsers which does not support HDR, but is currently the easiest solution right now.
A better approach is to use UltraHDR which is an extension to JPG. This has better backwards compatibility and will show a normal SDR image in most applications, and show HDR in supported ones. I will try to create a way to do this in the near future.
Description
FAQ
Comments (12)
I'm a total noob. Can you tell me exactly, step by step, what to do with that python script? ATM I'm generating high illuminated pictures
I assume you have a working Python installation from Comfy-UI or A1111 stable-diffusion-webui.
First open a command line from the folder you have saved the python script in, by shift+right clicking on the folder in Explorer.
write the following:
python mark-png-as-hdr.py path-to-input-file.png wanted-name-for-output.png
You will now have a file that if you open it in Chrome or Edge will now look correct.
@spillerrec It gives me the following
D:\sd.webui\system\python>python mark-png-as-hdr.py path-to-input-file.png wanted-name-for-output.png
Traceback (most recent call last):
File "D:\sd.webui\system\python\mark-png-as-hdr.py", line 19, in <module>
with Image.open(sys.argv[1]) as im:
File "D:\sd.webui\system\python\lib\site-packages\PIL\Image.py", line 3236, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'path-to-input-file.png'
@dart779 You where supposed to replace "path-to-input-file.png" with the path of the image you want to use. (If you shift-right-click on a file you get a 'Copy as path' option.) Maybe a quick introduction to the command line and how to navigate folders will help:
https://tutorial.djangogirls.org/en/intro_to_command_line/
However work is being done to add support in ComfyUI so everything is automatic, you could try to wait for that instead.
@spillerrec yeah maybe it's better XD as I said, total noob here. I've managed to run the script, had to run pip install image first, but apparently got no results whatsaever. I'll patiently wait for the automatized operation lol
Maybe I am not technical enough to make it from the description, but it does seem to have a desaturation effect on colours.
Looks cool, your examples! I could imagine this extension(lora?) would also be great for Flux... If I examine the workflow behind the pictures here I see the usage of your Lora, but not of the VAE? So please could you provide a comfyui workflow, or at least a link to a (simple!) comfyui workflow which could be used with your VAE in Order to get real HDR Images? I understand, I additionally have to execute your script after creating this picture!
I use stable-diffusion-webui and you set up the VAE in the settings menu, so you don't really need to do anything special. I don't use comfyui much, but I tried creating a sample workflow here:
https://civitai.com/images/80902239
There is a open pull request which adds support into comfyui so you don't need to run my script afterwards:
https://github.com/comfyanonymous/ComfyUI/pull/6651
You can check out these changes in different ways. I used:
git pull origin pull/6651/head
Then you need to make sure you are using at least pillow-11.2.1:
pip install --upgrade pillow
Make sure that pip command targets the correct python/venv that you use for comfyui. If this gives you issues, you can use my script instead and just use stock comfyui.
Note that there will not be any HDR in comfyui since HDR is not supported in HTML5 canvas, but it will render correctly in SDR and the saved file will be in HDR (if you updated pillow correctly).
I don't know which VAE FLUX uses, so I'm assuming they are incompatible. You would still need to train a LORA as well, since this is specific for each checkpoint (i.e. stock SDXL, Pony, Illustrious).
@spillerrec Thank you very much for your ComfyUI example provided here, it helped me very much as I could see, how it was intended to be used there! I mixed it together with mine and now I can also create some pretty cool HDR Images like you did - at least in SDXL now! It's a pity we don't have Flux-Dev and/or Flux-Schnell Lora/Vae's as Flux is very colorful in the generation - and I guess HDR would be a great boost! Unfortunately I don't understand enough about AI&HDR to train it for hdr. Also I don't have a powerful enough Card for this! Yeah, the open pull request would be cool - one step less(->which I forgot quite often, the last days!) A big thank you - for answering, doing the example, and doing your work with HDR, this is gamechanging as up to now, it was not possible - and on some AI generated Pictures it looks overwhelming in HDR on an (AM)OLED! I posted already some of my first tries with your Lora/Vae combination for you to see!
@CoolAIUser Awesome that you got it to work, your images looks great. They do a great job at showing how vibrant colors can look in HDR.
Ooh, very interesting! :o (Just wish the correction process wasn't so tedious...lol) I wonder if its possible to create a bat file that you can simply drag the image onto to process it automatically. I remember an early ESRGAN upscaler working that way!
Oh yay, I got it to work! XD