This workflow uses SAM to invert the colors of the character to be replaced. By doing so, it makes character replacement significantly more consistent that Minimax H3 alone.
How to use it
You must
(v2 and above) Install https://github.com/kijai/ComfyUI-SolAttn_triton if you want the speedup.
(v3) Install https://github.com/1038lab/ComfyUI-QwenVL for QwenVL video to text description.
(v1 only) Change the text prompt to describe both the scenario and the subject. Your goal is that Minimax H3 reproduces the original video as similar as possible.
Change the video and the picture, and make sure to change as well the resolution to match the original video.
(Optional, v2) change the SAM prompt until it masks only the subject to replace.
(Optional, v3) change the hint given to the Qwen node.
Why SAM and masking with H3
When using H3, it seems that the model will either recreate the original video as it is, without changing the subject at all (follows the <Video 1> too literally), or it will create a totally new video (follows the <Picture 1> too literally).
However, when inverting the color of the subject in the video, H3 is forced to produce a new video because the subject looks weird. So it can't follow the <Video 1> too literally anymore, otherwise it would produce a video with inverted colors.
Description
With QwenVL, describes the scene and the subject to do the replacement
FAQ
Comments (18)
Which version of Qwen VL should we be using for the Qwen-VL Mod node?
I tested with a few options and keep getting the following error:
AILab_QwenVL_Advanced.process() got an unexpected keyword argument 'video'
Specifically the first QwenVL Mod that does the video analysis.
The node I use automatically downloads the model I think. I don't remember downloading it manually.
@lomote The QWENVL model's censored though, it doesnt accurately describe the scene.
@mogmog I'm not sure which VL models are not censured, so feel free to suggest a drop in replacement.
@lomote I think there's a built in node with the same name, because it wasnt orignally flagged as a missing node. I did install that version and tested again with the same error.
I've tried a few different videos and Qwen models from the drop down.
I've rarely had luck with the qwen llm nodes, so maybe its something on my backend (nightly comfy and backend python should be up-to-date via the .bat file), but thats the only error im getting.
I think I figured it out. For some reason the qwenvl mod node was not updating (noticed it only had 2 image inputs when i tried to re-add the node). Blew away both the Comfyui-QwenVL (and Qwen-VL Mod) folders and reinstalled with: pip install --upgrade --force-reinstall <package>. Appears to be running now.
If we didnt want to use the reference audio but generate new audio instead, how would we rewire it?
This has worked really well for face/head swap.
Just change the SAM prompt to "The woman's head". I also recommend adding this enforcement throughout the generation prompt.
Example (insert features/anatomy you want to stay identical):
The target video replaces only the head and face of the woman in <Video 1> with <Subject 1> while fully preserving the entire body below the neck, the exact _______, the low-angle camera, the environment, and the lighting.
Hi is it just me or what background not being copied from video, movemnet in dance is just being "generalized" not copied. Some movemnet are just being ingnored and model replace it with something else. Bad prompting or other issue?
The last example of the dance didn't copy exactly, and there are artifacts. With other simpler scenes this works better.
Here the prompt was produced automatically with QwenVL. I don't know if it can be improved further, but if you get a prompt let me know.
@lomote I think we need something like fine-tune with better conditioning - maybe like SCAIL-2?
great stuff, the only problem right now is liquids hitting the face, with invert the model just makes a blue ink for any white liquids, there has to be a way to fix it but I cannot figure out.
Really like this workflow, especially the SAM invert trick.
You can add KJNodes Model Preview Override for sampling previews, makes it a lot easier to keep an eye on longer H3 runs: https://github.com/kijai/ComfyUI-KJNodes/blob/main/nodes/preview_override_node.py
Also found a nice speed win: added a second Scale Image to Total Pixels right after Image Composite Masked, before the reference video goes into H3, and dropped it from 0.5 to 0.2 megapixels. Sampling time went from 3:01 to 2:01 (roughly 1.5x), total workflow time from 280s to 212s. Compared both outputs 1:1, motion and detail included, barely any difference.
[edit]: Saw you already mentioned this, might be worth just adding the node in for people to tweak as needed.
One more thing that could help: an optional pause after QwenVL and before H3, to check or tweak the prompt before the expensive sampling starts. Would be nice if it's opt-in so it doesn't slow down a normal run.
I'm newish & don't understand; can you provide a workflow please?
for live preview: install the node from the link above and wire it as Load Diffusion Model > Model Preview Override > Turbo Lora.
for the 1.5x speed boost: clone "Scale Image to Total Pixels", place it between Image Composite Masked and MiniMax H3 Reference to Video, and set it to 0.2
It really sucks that the inverting hack needs to be done to reliably swap a character, otherwise the model just completely ignores one's prompting. For a 'state of the art' model that everyone claims this thing is, it seems like kind of a joke that it can't just do this natively. And yes, I know all about how the r2v prompts are supposed to be structured in h3; it usually doesn't make a difference, even with a perfectly structured prompt. Unless the characters you're swapping are literally visual opposites from each other, the model is just going to keep the original subject no matter how clear your prompt is...
Anyway, venting over; thanks for the hack. Glad it's working out for people.
Awesome, what if your SAM3 patch could do a full motion transfer? (Image background stays). Anyways. Left you a large sum - don't spend it all in one place XD :)