WORK IN PROGRESS
not VideoJam not Meta - only an inspired loRa
LoRa to improve the performance of Hunyuan video against the critical issues (of current video models) exposed by VideoJam.
Description
FAQ
Comments (30)
It looks like this LoRa has some great potential
🔥🙌
I was thinking VideoJam was going to be a node, but if this works thats awesome. Any one that hasnt heard about it the tldr is free update to all video models at no gpu cost to render movements with much better consistency (very visible with humans going upside down or spinning)
I was hoping so too. I don't think I can definitely achieve the results of videojam, but I want to get inspired by it to better understand how the models "think" about movements
cant wait to see how it work.
i would love to know how this lora was made in details 💗
🙌🔥I'm collecting clips from a few frames to try to correct some dynamics, in this beta I trained 40 clips mostly with respect to human physicality. At the moment a later version is being trained where the dataset is 200 clips. at the training level, no different in technique from others, simple prompts were provided, like those in the example on the VideoJam page, so this time I did the captioning by hand
This isn't the actual videoJam from Meta, but just a normal lora trained on critical poses?
is just a normal lora trained on critical poses and actions 🙌 No Meta was harmed in the creation of this loRa 😁
Science has still not solved the hula hoop. But very cool project. ;)
Thank you! 🤣🙌
any explanation of what this lora was actually trained on?
🙌video clips and images of physical dynamics and all the critical issues shown by the tests of the proprietary models
good potential but massive block artifacts as currently seem to be seen with most loras. The grass on the goat example is what i'm referring to, thats pretty bad.
thanks! I'm trying to figure out how to fix it, probably the next versions will be driven from scratch🙌
@Ciro_Negrogni This isn't caused by the lora, but by the method used in the workflow. I had the same issue for a long time but managed to rectify it.
I've just posted an example video (dragon) to show how it can look.
@galaxytimemachine nice
@galaxytimemachine how did you managed It?
@galaxytimemachine Yes, give up the secret, c'mon, don't make us beg.
The workflow is linked to the videos, but I'm using v1.4...coming soon!
@galaxytimemachine @0C00l I do know it needs to be managed with blocks, there are double_block loras that can be used but this depends on how it was originally trained. Just setting double_blocks could remove all motion from the lora for example
looks very promising I will test out!
🙌🙏
Does it work with anime visuals? Thanks.
the model is trained on realistic material, so I don't think it works well, maybe with a lower weight it would be worth trying
@Ciro_Negrogni Thanks mate.
Any progress on this?
🙌🔥as soon as I get to version 0.7 I'll publish the stable 0.5
@Ciro_Negrogni Awesome, thanks for the heads up man.