Yidhar psd-layer-vae — An RGBA Autoencoder for Editable Layered PSDs
⚠️ Please read before downloading: This is a standalone VAE trained from scratch (f8c32: 1/8 resolution, 32-channel latent). It is not compatible with SD1.5 / SDXL / Flux or any other existing diffusion model, and cannot be used as a drop-in VAE replacement in WebUI or ComfyUI. Use the psd_layer_vae inference code included in the repository. It is intended for encoding/reconstructing layered images, RGBA data processing, and as a latent space for training layered generative models.
What it is
Each layer is encoded into its own latent. At decode time, all layers of the same artwork are decoded jointly with cross-layer attention, producing straight (non-premultiplied) RGBA. An HDR channel can reconstruct linear light brighter than SDR white. About 82M parameters in total (34.8M encoder + 47.2M decoder).
Key strengths
1. Clean transparent areas — no gray haze when stacking layers
Existing RGBA VAEs often leave a faint alpha residue across the whole layer. Invisible on a single layer, but stack dozens of them and you get a gray veil; saved as an 8-bit PSD, it also breaks layer bounding boxes. This model outputs essentially zero in transparent regions:
Average over 72 line-art layers (h3 vs Qwen-Image-2.1):
Alpha soft IoU: 0.958 vs 0.578
Mean residual alpha in transparent areas: 0.02/255 vs 0.80/255
2. Accurate thin lines, semi-transparent strokes and dense text
Text on transparent layers: 42.06 dB PSNR (Qwen-Image-2.1: 29.82), alpha soft IoU 0.945 (vs 0.745).
3. Decode an entire PSD at once — faster, with less VRAM
Compared with per-layer Qwen-Image-2.1 VAE decoding (H200, bf16): 2.5–3.7× faster decoding at about 1/3 the VRAM of its batched decode. At 1024² × 32 layers, Qwen's batched decode runs out of memory; this model completes normally.
4. Higher per-layer and composite fidelity
On 20 held-out PSD windows (15.7 layers each on average): per-layer PSNR 53.96 vs 41.74; composite PSNR 37.00 vs 34.97.
5. Strong on regular images too
Leads both Qwen-Image-2.1 and FLUX.2 VAEs in PSNR / SSIM on every test set: ImageNet, LAION, photos, illustrations and danbooru.
6. HDR linear light support
h3 (recommended) — for editing and exact per-layer reconstruction. Best pixel fidelity.
g3 — for sharper decoding in generation. About half the rFID of h3, ~0.3 dB lower PSNR.
h3 and g3 use different latent spaces — do not mix their encoders and decoders.
Quick start
import torch
import psd_layer_vae as P
model = P.load('Yidhar/psd-layer-vae', 'h3') # or 'g3'
rgba = ... # (num_layers, 4, H, W), bottom to top, straight RGBA, sRGB, [0,1], H/W multiples of 8
latent, box = P.encode(model, rgba)
out = P.decode(model, latent, box)
image = P.composite(out['rgba'])
See the HuggingFace page for full usage (HDR, layer order, fp32, etc.).
Known limitations
Perceptual quality (LPIPS) still trails Qwen-Image-2.1 and FLUX.2. On rFID, g3 beats Qwen-Image-2.1 but not FLUX.2.
Joint decoding causes slight cross-layer crosstalk: after editing or deleting one layer and re-decoding, the other layers change by about 0.12/255 on average. If they must stay bit-identical, re-decode only the edited layer and keep the previous output for the rest.
Extremely bright HDR scenes (highlights above ~10 stops) still fall noticeably behind in some cases.
g3 details look realistic but are not identical to the source; use h3 when exact reconstruction matters.
Thanks to comfy.org and others for providing computing resources.
Links
HuggingFace: https://huggingface.co/Yidhar/psd-layer-vae