In this tutorial, we will learn how to animate images using LTXV, specifically those that initially do not yield good results.
The workflow used is a customized modification of the following workflow:
https://github.com/sandner-art/ai-research/blob/main/LTXV-Video/ltxvideo_I2V-motionfix.json
This approach was inspired by the idea of using blur effects to improve results, as proposed by VoidVisionary in:
https://civarchive.com/models/1057138
The zip includes: The workflow, a .png image of the Mona Lisa without blur, a .png image of the Mona Lisa with blur and a .psd archive with the edition.
This workflow incorporates a modified version of the VAE created by SpacePXL, which you can find here:
https://huggingface.co/spacepxl/ltx-video-0.9-vae-finetune/tree/main
In this case, I am using ltx-video-v0.9-vae_finetune_all.safetensors, which you should place in the models/VAE folder of ComfyUI.
The model used for LTX is ltx-video-2b-v.0.9.1.safetensors, which should be placed in the models/checkpoints folder of ComfyUI:
https://huggingface.co/Lightricks/LTX-Video/tree/main
The text encoder used is t5-v1_1-xxl-encoder-Q6_K.gguf:
https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf/tree/main, which should be placed in the models/clip folder of ComfyUI.
This workflow utilizes the following custom nodes:
Once everything is set up, we will load the initial image of the Mona Lisa. If we use it as is (without blur) and render it, we will get a very limited animation. To enhance the animation, we will simply use a image editor, select the face, copy it, paste it onto another layer, and apply a slight horizontal motion blur. I’ve provided a Photoshop file where the face has been converted into a Smart Object with a horizontal motion blur of 12 applied. I also add the image of the Mona Lisa already blurred.
If we now copy and paste our modified image into ComfyUI, we will see a much smoother and more natural animation.
This technique can be applied to almost any image, and experimentation is encouraged.
Description
FAQ
Comments (12)
Wth why does this even work??
As I understand it, the workflow has three phases:
In the first phase, we add some motion blur: The model has been trained with videos where the parts containing motion blur show more movement. By adding motion blur to the face, we prepare the model to animate the character's face.
In the second phase, there is a general compression of the image: This destabilizes the image and generates new random noise, making it easier to animate.
In the third phase, we attempt to stabilize the animation: The goal is to ensure that everything we’ve added to the image does not lead the model to hallucinate the entire animation.
@dumdidum No motion blur is added in this workflow in the 1st stage. It is a single image resized, then upscaled, then packed into a video with a single frame
@joesixpaq If you read the text you will see that adding motion blur to the the faces in an external software is an important part of this tech.
Please post an HD screencap of the workflow. I like to know what I will be getting before I download.
Finally some no bs, overcomplicated workflow. Works great!
Just did a quick test using GIMP to blur the Monalisa myself and it kind of worked... blurring just the face just the right amount gives motion in the generated video.
Blurring the entire image resulted in no motion.
I set blur in GIMP to
Rate:100
Enable Smooth Stroke
Quality: 44
Weight:442
Then with a face sized brush (325)
Brush over the face 6 times... the bruising motion direction changes the motion you might see, so it is kinda hit and miss still...
Why not use the blur node in ComfyUI? If I find the time, I'll try that. Also masking the face should be a easy thing to do with SegmentAnything or similar...
got error:
SamplerCustom
'original_shape'
how fix it ?
Using this on Linux, I get the following error.
SamplerCustom
'original_shape'
how can i fix it ?