Used https://civarchive.com/user/seruva19 's Ghibli dataset. 120 images, batch size 4, 100 epochs, 3000 steps. Learning rate: 1e-5. 13 hours running on 2 x rtx 5000 ada gpus. I used diffusion-pipeline since it allows distributing loads across multiple gpu's.
*Update V2.0 I doubled the lr rate to 2e-5 to make a more potent LORA.
Image2Image guidance: If you want to do img2img, I recommend setting sampler = lcm and scheduler = linear_quadratic. The denoise will be between .3 - .4 and steps around 5-10. Will need to adjust on a case by case basis.
Description
I upped the LR to 2e-5 (double the previous) and the training a bit longer 112 epochs. This Lora is much stronger than the previous one.
FAQ
Comments (4)
Alas, LoRa should be good, but it doesn't work on my model 🙀(hidreamfulldevfast.safetensor model)
Which model are you running? I have tested it on full and dev, Q8 and F16. Are setting the trigger word(s)? The strength should be 1.
@rusty2930 HiDream I1 FULL-DEV-FAST - and so many error on cmd comfy (ERROR lora diffusion_model.double_stream_blocks) 🤷♂️
@Kotoshko Double stream blocks are how the model/lora wires both the text and latent image together. In comfy are you wiring the clip AND the model into the LoRA, then using that output to process the prompts?