UPDATE: You can download SVD ver 1.1 here: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt-1-1/tree/main
"This model was trained to generate 25 frames at resolution 1024x576 given a context frame of the same size, finetuned from SVD Image-to-Video [25 frames]."
From Stable Video Diffusion's Img2Video, with this ComfyUI workflow you can create an image with the desired prompt, negative prompt and checkpoint(and vae) and then a video will automatically be created with that image.
v1.3_sd3: txt2video with Stable Diffusion 3 and SVD XT 1.1
For information where download the Stable Diffusion 3 models and where put the .safetensors: https://civarchive.com/models/497255/stable-diffusion-3-sd3
v1.3_cascade: txt2video with Stable Cascade and SVD XT 1.1
For information where download the Stable Cascade models and where put the .safetensors: https://gist.github.com/comfyanonymous/0f09119a342d0dd825bb2d99d19b781c
v1.3: svd default is now: svd_xt_1_1.safetensors
.gif creation instead .webp(.mp4 is still present)
v1.2: changed the default value of fps svd img2vid_conditioning from 8 to 10, previously it was erroneously different from the save value and save webp value of 10
v1.1: more clear interface
v1: first version
Required:
1- on Windows: 8+ GB VRAM NVIDIA gpu only
on Linux: 8+ GB VRAM NVIDIA gpu and AMD gpu
2- ComfyUI: https://github.com/comfyanonymous/ComfyUI
3- this worfklow
4- in ComfyUI\models\checkpoints:
https://huggingface.co/stabilityai/stable-video-diffusion-img2vid/blob/main/svd.safetensors <-- for 14 fps video
or
https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/blob/main/svd_xt.safetensors <-- for 25 fps video
5- a checkpoint in ComfyUI\models\checkpoints
6- a vae in ComfyUI\models\vae (optional)
7- ComfyUI Manager for the Missing Nodes: https://github.com/ltdrdata/ComfyUI-Manager
P.S.: The recommended resolution for SVD is 576x1024, but in the workflow I set a default value of 512x768 (with 40 steps, 20 steps is good enough). It's up to you to choose the resolution and steps that you want... but 512x768 with 40 steps is a good starting point for your attempts.
If you want to support me, check out my Patreon: https://www.patreon.com/Shiroppo
Follow me on civitai(!!!): https://civarchive.com/user/shiroppo
Follow Me on Twitter/X: https://x.com/ShiroppoTwit
Pixiv: https://www.pixiv.net/users/96490888
---
Description
- svd default is now: svd_xt_1_1.safetensors
- .gif creation instead .webp(.mp4 is still present)
FAQ
Comments (12)
Got it working, interesting stuff. Are there more tips or a more comprehensive guide on using this? How do you control the length of the video? Your default settings has 25 frames at 8 fps in the img2vid_conditioning node but 10fps in the save and video combine nodes, any reason why they differ? Is there any way to control what the video does? I generated one of a woman on a street and the video was a static picture of the woman with a background moving around her. Another was just a static pan from left to right. Interesting but a bit strange.
https://stability.ai/news/stable-video-diffusion-open-ai-video-model <-- official svd page
- 8 fps in the img2vid_conditioning it was my mistake during the initial experiments, corrected in v1.2 with the value of 10, thank you very much
- the length of the video is frame / fps so the default value is 25/10(in save, save webp and conditioning)=2 seconds and half... if you try 5 fps you will have a 5 second video, but above 4 seconds the quality drops a lot, unfortunately this model svd is only made for very short videos, I hope stability ai will create new models in the future for longer videos
- for the prompt, you have to be precise about which part of the body to move, an example is "1girl, wind on hair", that's also why a few days ago I had thought about making it easier for example making a txt2img2img2video, with 2 prompts instead of 1 but the results were not good, for now writing a prompt+an action is the only solution
As far as the frame length like the link said it's pretty restrictive right now, the models are trained for 14 frames with regular SVD and 25 frames with SVD-XT.
The (untried) solution to that limit would just be to save a sequence and just use one of the looping nodes and a conditional to feed the last frame of the video the model generated back into another img2video to produce another 25 frames and repeat as desired.
I tried doing the feedback of the last image into another ksampler thing finally with an img2video... as long as your last frame was stable (sometimes it goes weird on you) it'll usually keep generating smoothly. If there's a ton of motion it might bounce back in a weird way or start doing something else but if the image is similar enough to the first it'll work. So keep motion lower maybe? It would be harder with txt2video since you can't easily inject the prompt conditioning into the conditioning the image produces. I just kept getting mismatched tensor sizes regardless of attempts to forcibly resize the conditionings. It's not predictable but it's fast enough that I don't mind. I feel like this process would work better with animatediff video2video where the image feedback from the first sampler probably wouldn't even be required so it could be set up as a loop more easily and allow for more optimal batch sizes for your GPU to be run with the same results even if you kept the length the same.
I used qq-nodes-comfyui's "image accumulator start" and "image accumulator end" to gather all the images produced, set count to 49, fed the first 24 of the batch from ksampler 1 into that, the last one into a clip setup for the second ksampler and then fed those 25 into the accumulator node which trips the count and sends it through the end node (needed a cr image list to batch, nothing else worked for some reason) to the animation output.
Very cool, Just tested it myself and it works super well
could do make the sampler of generating video being lcm
I keep getting this error. Can you help me?
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "D:\AI\ComfyUI\execution.py", line 153, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\AI\ComfyUI\execution.py", line 83, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\AI\ComfyUI\execution.py", line 76, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\AI\ComfyUI\comfy_extras\nodes_video_model.py", line 45, in encode
output = clip_vision.encode_image(init_image)
AttributeError: 'NoneType' object has no attribute 'encode_image'
With google I found this: https://www.reddit.com/r/comfyui/comments/184d7s4/cant_get_stable_video_diffusion_to_work/
Thanks that's exactly what was lacking for img2vid
RuntimeError: Conv3D is not supported on MPS
How to solve this problem? My computer is a new type of Mac computer
With google I found this: https://github.com/comfyanonymous/ComfyUI/issues/2044
@shiroppo OK thanks👌
