I can tell you that right now you have two main options:
( https://github.com/tdrussell/diffusion-pipe )diffusion-pipe came first and is more mature but is heavier VRAM wise. People with 24GB like your 4090 have successfully trained with videos using it. I have 16GB and only ever was able to train with images. It does have an option for using multi GPU but it doesn't work for Hunyuan yet. I think that's what trained this particular LORA based on it's formatting and metadata(or lack of).

( https://github.com/kohya-ss/musubi-tuner )musubi tuner by Kohya-SS of sdscripts fame is a bit newer but has more options for those with less than 24GB of VRAM like myself(block swap, split attn, etc). I trained a LORA last night on 768x432, 64frames @ 24fps videos successfully with 16GB VRAM!

As far as settings the defaults for either are a good place to begin. Both do have FP8 mode so be sure to make use of that, the activation memory requirements of video are HUGE so you need all the free VRAM you can get for that. With Musubi you can use --split_attn to help with that and --blocks_to_swap(up to 36 works for me) to help further. I've only managed a few experiments myself and can't really speak further. Hope that's helpful

Edit: A few other small things:
--LORAs trained on only images do work, but often have stiff or unnatural motion. Lowering the LORA weight can help a tad though.
--Most of the LORA on civit were trained with diffusion-pipe as of the writing of this comment on January 9, 2025, but I think eventually more people might use musubi given it's lighter requirements(unless d-pipe implements more memory saving options in the future!) Even those with higher VRAM can benefit from training at higher res or longer frames.
--Not specifically training, but if you wanna use multiple LORA together during inference(for instance character + action), it's often beneficial to use the block edit node from HunyuanWrapper to apply them only to the double blocks. They will play much more nicely together.

@blyss for anyone reading this if you are using Hunyuan in native Comfy instead of the wrapper, there is another node called "Hunyuan Multi Lora" or similar that allows you to turn on double blocks. It was only added 3 days ago.

@blyss Thank you so much for all of this helpful information! There's a whole lot here that I need to dig into, and I appreciate all of the advice!

i have yet to try it, but what other objects work well? can you try some super weird stuff too?

@crombobular Is a traffic cone weird? Because that works for some reason.

@Monster lmao yeah

@crombobular mushrooms work well, too ;) you can even make the mushrom cum inside and then stuff is dripping out ;D

@definitelynotadog I keep an eye on musubi-tuner but so far no lulck in running it, it seems i'm always missing some files or settings, anyway thanks for the .toml

f

f

learn comfyui and use huanyuan nodes