⚠️ The I2V Vid extend workflow is highly experimental and requires high end GPUs.
I've created this Wan Video Image2Video workflows based on Kijai's workflow and the Native ComfyUI workflow, added an easy to use interface along with upscaling and frame interpolation.
Key Features:
1st pass - generating vanilla video
2nd pass upscaling using user selected upscaler and frame interpolation to 32FPS
This workflow comes pre loaded with my Wan14B RunPod template:
https://civarchive.com/articles/11960/wan14b-in-1-click-with-workflows-included-runpod-template
Join my Discord server for updates on new LoRAs, tips & tricks, Workflows, RunPod templates and the holy right to be close to an amazing charismatic person like myself:
https://discord.gg/fyha5Pzm
Description
Added TeaCache support for the non native ComfyUI workflow - generation should be much faster!
Removed Kijai's clip encoder and added the native ComfyUI clip encoder (split_files/text_encoders/umt5_xxl_fp8_e4m3fn_scaled.safetensors)
download it here:
https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/tree/main/split_files/text_encoders
FAQ
Comments (41)
New WF Error.
Failed to validate prompt for output 80:
* LoadWanVideoClipTextEncoder 217:
- Value not in list: model_name: 'open-clip-xlm-roberta-large-vit-huge-14_fp16.safetensors' not in ['clip-vit-large-patch14\\model.safetensors', 'clip_l.safetensors', 'llava_llama3_fp8_scaled.safetensors', 'oldt5_xxl_fp8_e4m3fn_scaled.safetensors', 'open-clip-xlm-roberta-large-vit-huge-14_visual_fp16.safetensors', 't5xxl_fp16.safetensors', 't5xxl_fp8_e4m3fn.safetensors', 'umt5-xxl-enc-fp8_e4m3fn.safetensors', 'umt5_xxl_fp8_e4m3fn_scaled.safetensors']
Tried 3 of the text encoders shown and same error.
it happend to me the same just download this vae "https://huggingface.co/Kijai/WanVideo_comfy/resolve/346ea0b6848edd2aa7e34d0444b2b05ebc7bd97a/Wan2_1_VAE_bf16.safetensors?download=true
youll be fine
Kijai removed that clip from his HF repo.
I updated the workflow, download it again.
Download the new clip here
https://huggingface.co/Kijai/WanVideo_comfy/blob/main/open-clip-xlm-roberta-large-vit-huge-14_visual_fp16.safetensors
@Hearmeman Giving it another go! Thank you.
@Hearmeman Now it's failing on the diffusion model. I might not be able to run it on 12GB. I probably will have to use a GGUF.
How do I obtain the missing node #87 (upscale factor)? Thanks
comfyui-kjnodes
Outputs from the native workflow appear to only be affected by the prompts, not the input image. Is this true for everyone?
The new workflow is unusable cause there is no source on the web for open-clip-xlm-roberta-large-vit-huge-14_fp16.safetensors
Nothing works. My 32Gb RAM and 24Gb VRAM are at 100% and the generetion is stuck at 0% by 30 minutes.
I have the same problem. Getting nearly 60s per iteration. Not sure what I'm doing wrong.
Use quantized FP8 models. @StanleyPain @Kanginak
@StanleyPain The fact is the user who made this workflow just told me he uses 48Gb of VRAM and nearly 100Gb of RAM. And they think normal people could run their hacker stuff...
@Kanginak I run on RunPod (cloud platform)
You don't need any "hacker stuff", you only need to spare some extra cash if you want to run the latest full sized models.
This is a new model that was released in the past few days, there are already quantized models for people who want to run locally with lower end GPUs.
@Hearmeman So you're saying this isn't meant for 24GB cards?
@StanleyPain 24GB cards should run fine, if you’re running into memory issues, lower the resolution/video length and use a quantized model
Any way to get this working with 16GB of VRAM? I run out very quickly.
I even tried lowering the resolution by half. I found another workflow that works, but I prefer this one due to sage attention.
@_degenerativeai_ Which model are you using?
@Hearmeman wan2.1_i2v_480p_14B_bf16.safetensors. I've been able to successfully use this model as well as the 720p version in another workflow, but they are a bit less complex and don't use sage attention.
@Hearmeman I figured out the issue. By default the workflow had quantization turned off in the model loader node. Once I turned that on it worked perfectly.
@_degenerativeai_ That's not an issue, that's how I designed it.
Quantization hurts the quality, of course everyone can feel free to disable or enable it.
@Hearmeman For me on 16GB mobile 4090, without quantizing I run out of VRAM. I’ll upload one of my results. That is good for me. I’d prefer not to use quantization, but I have no option.
@_degenerativeai_ Use a GGUF model variety, download the Q8 and work down from there. There's a 720P version and a 480P version. I have a Q5 GGUF model running on my 3070 with no problems of OOM
@woodenpickle I appreciate the response, but I would prefer quantizing over using the GGUF model. I feel like the quality will be better overall.
When I load the workflow, in the "WanVideo TeaCache" node, the option called "end_step" has the value NaN. What am I supposed to enter here?
Right click and reload node
Sigh... No matter what I do, the needed nodes are always conflicting in the manager... Never succeed in using any of the workflows here on Civitai. Any suggestions on how to install a working comfy that accepts the needed nodes?
Update comfy via comfy manager. Use comfy manager to install missing nodes. Follow the instructions laid out in the description of this workflow. Follow links and download whatever nodes comfy manager doesn't have via git pull to your comfy nodes folder. It's like installing anything else for Automatic1111, it's not that difficult, just follow the steps and you'll be fine.
@woodenpickle thanks for your answer...! I think I did all that, but the way you write your answer makes me think I need to try again (tried many times and the nodes keep conflicting each other).
@jay_rich Sometimes the nodes come pre-loaded with the correct file name but you need to manually select it in the drop down anyway I've noticed or it will still be a red node. Overall my opinion is that whoever designed comfy should be taken out behind the woodshed. This is the least intuitive or accessible piece of software designed solely so that tech bros can beat each other off about being "hardcore" about AI (sorry, I hate it so much).
You could try SwarmUI which is overall miles better than ComfyUI in most regards sigh EXCEPT video generation right now. McMonkey is fantastic but the video gen settings are a mess. There are just too many models and too many settings and he has tried to bundle them all under a single unified "video settings" which means you have about 15-20 different toggles, numbers, sliders, etc and only 3 or 4 of them are pertinent to your individual model. Then you have to know what the recommended settings are for your individual model which isn't loaded by default like it would be in a comfy workflow. So anyway there's my morning rant. I'm going to drink coffee and go make some images because video right now is kind of garbage anyway unless I want to make yet another nude lady looking at the camera or spend 3 hours waiting for a glitchy video that doesn't adhere to my prompt or look like my original image at all.
@aikage Hahha... Awesome! I share you view on Comfy! The massive potential it has, and then all the mess. A1111 never complaints, but it lacks most of the possibilities of Comfy.. ... Think I will go and get a coffee and start todays fight on trying to get Comfy to work, so I can make a nude lady staring at me convincingly :D ;)
@jay_rich The best approach that works for me is setting up an environment for each use case.
This is not ideal, but cuts boot time and avoids conflicting nodes.
If you can be more specific about your problem with my workflow I can try to help.
@Hearmeman Thanks for offering to help.. I just made a complete fresh Comfy install (should I use a special build or something?). Added the manager and started. Then updated comfy to latest and loaded your workflow with lora support. Next added all missing nodes from the manager and made a restart. Then, when starting comfy again, I get new missing nodes warning. going again to missing nodes, these are still there with import errors:
ComfyUI_PuLID_Flux_II (also has conflict nodes: ApplyPulidFlux, PulidFluxEvaClipLoader, PulidFluxInsightFaceLoader, PulidFluxModelLoader),
ComfyUI-Logic,
ComfyUI-PuLID-Flux-Enhanced
All three nodes has these buttons with "Try fix", "Switch version" etc. Have you any idea on how to fix this? Thanks
@jay_rich PuLID is notorious for having issues.
The workaround I made in my Flux runpod template is running the following every time the system starts
python3 -m pip install facexlib onnxruntime-gpu timm onnxruntime
This solves my missing nodes issues with PuLID.
Sorry to say it, but your pod setup doesnt work either. And saying either I purely mean in my situation. Not everyone else. So no blaming. Just saying that it doesnt work. The env vars in your video are different, so tried with both setting the ones that are there to true (and wait two hours for the used storage meter to rise - note: created the drive as you recommended). And when that setup did not work, i tried adding one of the vars (I2V), the env vars. Didnt work either. The 8188 just keeps being yellow. Waited all day, and the money balance just keeps on degrading :O Any advice? Cheers
@jay_rich I'm constantly pushing updates and things change, the pinned post in the Youtube video covers the changed environment variables.
You should monitor the logs and see if something is wrong and how long you have left for the models to download.
You shouldn't change the environment variable name, just the value from false to true.
If you're still having trouble, send me a DM I'll help.
@Hearmeman Yea.. I bet it is hard to keep up with updates... But really nice of you to help. I actually just now succeeded in starting the pod, but the template is not opening in comfy... I will investigate myself first, and if I cant get it to work, I will write you.. Thanks again! :)