I created this workflow that supports the native ComfyUI implementation of Wan Fun with multiple ContorlNet support.
Short bullshit free tutorial -
Key Features:
Multiple ControlNet options (Pose, LineArt, Depth)
Initial frames and suggested resolution to create the perfect initial frame to animate
Upscaling and frame interpolation to 32FPS
Description
FAQ
Comments (37)
Would you be able to do a tutorial where the control net takes just an still image and uses it to make sure the character stays coherent throughout the generation especially the face features
For T2I?
@Hearmeman is this possible to do for i2v?
@IonBlaster7 Why do you need ControlNet if you're using I2V?
@Hearmeman he obviously wants to do deepfakes of some sort
@sunburnt_toasty Wish them the best of luck with that, I don't support this.
@sunburnt_toasty not at all I was trying to get my character to stay coherent through my i2v but adjusting Lora weights and changing the styling prompt fixed my issues don’t need it anymore :) I was trying for a semi realistic 3d animation style for atom Eve but it kept warping her face to look like someone else halfway through. Through trail and error I worked it out
@Hearmeman inreally appreciate your tutorials thanks for helping all of us out here ❤️
Thanks for the workflow!
Have you seen differences in quality compare to the regular Wan model for 720p version and duration? My 3090 seem to cap at 81frames before OOM.
Also, I like how you can use number to change the controlnets. Sometimes multi-controlnets helps in certain case, so it would be neat to have the possibility to use 2 or all of them.
For the initial image, I feel like it's highly necessary to have it at the same position as the ref video, so perhaps in another version, have a bypass option to automatically generate the pose from the ref image (which you've started bellow) and connect it to a new ref image for the video?
Awesome work!
Thank you!
I didn't really find a purpose to use 2 control nets but I haven't experimented enough.
This workflow is getting quite a bit of traction, I released it as a fun project and didn't think it would get much attention so I might give this some more work.
I'm also planning to do the initial frame generation within that workflow.
Thanks again
@Hearmeman There aren't many Wan Fun workflow out there yet (like 3), so that would explains why.
For myself, I find multi controlnets helps lowering the diffusion animation of large amount of details in the background with depthmap / lineart and open pose for character movement.
Regular Wan can be pretty hard to have control on some i2v generation, so controlnets seems to be the solution!
@Catz: try this: https://i.imgur.com/hEMcmsz.jpeg
I am not finding WanFunControlToVideo node. I tried searching for it and I am not having any luck.
Here it is: https://github.com/comfyanonymous/ComfyUI/blob/master/comfy_extras/nodes_wan.py#L53
If you don't have it in your installation, you need to update comfy. It was added in a commit just a week ago.
https://github.com/comfyanonymous/ComfyUI/commit/3661c833bcc41b788a7c9f0e7bc48524f8ee5f82
Thanks for the quick response. Yeah, it's strange. I saw that it was a part of comfy-core in my research but it was not in my installation. I updated to latest using the manager and it still wasn't there. I even went as far as installing the new windows desktop app with auto update turned on and it couldn't find it either. I will try installing directly from github.
@lewisayers252 Flow said a few days ago it's in the nightly build, so that might still be the case.
Fireman is correct, updating to nightly fixed it for me.
What is the max length of uploaded video?
Seems to work great so far. Not sure how good the quality can be with 1.3B but at least it's fast for testing purposes. Thanks for sharing!
Why does it save like 6 videos of all the different qualities? Seems unnecessary
This may be a stupid question, but I don't see anywhere in the workflow on how to lower the contronet strength. I am having issues where the control is overriding my image and prompt, creating a horrid output video with both the control video and image characters blending together into a jumbled mess. Both the Image and video used are near identical in appearance and poses, as stated in the tutorial video you provided.
Looks useful but I keep getting this error
RuntimeError: mat1 and mat2 shapes cannot be multiplied (769x1279 and 1536x1536)
My starting video is 1920x1080, I use the 832x480 in the resize, and my starting image is 1344x768
Any ideas? Thankyou
this usually means there is a model mismatch of some type, or at least I get that sort of error when it is. Ensure you are using models (checkpoint, clip, vae) that are compatible with each other.
Question: Does wan have a simple v2v functionality by some means, like hunyuan with denoise < 1 works?
All the V2Vs I've seen are controlnet based, and when I tried the above trick it just corrupted the vid completely.
for some reason whenever i run this i get no errors but the output is just a gray static video. anyone know a fix?
try denoise 1.0 / cfg around 7
check your clip skip setting, if wrong you'll get that. i usually run -1 for Wan, -2 for most SDXL
I had error:
Warning: Missing/Unavailable Node Types
When loading the graph, the following node types were not found or unavailable:
PathchSageAttentionKJ
No selected item
Nodes that have failed to load will show as red on the graph.
update ComfyUI so you have the Manager button. Install missing nodes in workflow option. easy peasy.
Sorry, so useless and way too much nodes for the purpose... Yes, just for experts... but, so what..
I could have made it super barebones and send you to another workflow to extract the first frame and then you'd still complain
In the 1st pass process, the generated images are only gray and static, without any errors. How can I solve this problem?
DOSENT WORK GRAY VIDEO
cannot import name 'sageattn_qk_int8_pv_fp16_triton' from 'sageattention' (C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\sageattention\__init__.py)
I don't understand comfyui tutorials. You need to install 75 different things in order to make this workflow work and non of them are ever mentioned.
update ComfyUI so you have the Manager button. Install missing nodes in workflow option. easy peasy.
very nice workflow, but unfortunately, all I got is a grey flickering video. anyone know how to fix this?