This workflow uses SAM to invert the colors of the character to be replaced. By doing so, it makes character replacement significantly more consistent that Minimax H3 alone.
How to use it
You must
(v2 and above) Install https://github.com/kijai/ComfyUI-SolAttn_triton if you want the speedup.
(v3) Install https://github.com/1038lab/ComfyUI-QwenVL for QwenVL video to text description.
(v1 only) Change the text prompt to describe both the scenario and the subject. Your goal is that Minimax H3 reproduces the original video as similar as possible.
Change the video and the picture, and make sure to change as well the resolution to match the original video.
(Optional, v2) change the SAM prompt until it masks only the subject to replace.
(Optional, v3) change the hint given to the Qwen node.
Why SAM and masking with H3
When using H3, it seems that the model will either recreate the original video as it is, without changing the subject at all (follows the <Video 1> too literally), or it will create a totally new video (follows the <Picture 1> too literally).
However, when inverting the color of the subject in the video, H3 is forced to produce a new video because the subject looks weird. So it can't follow the <Video 1> too literally anymore, otherwise it would produce a video with inverted colors.
Description
Outfit replacement
FAQ
Comments (20)
Can you provide guide for the prompt on how to use this workflow?
Please take a look at version 2. It is important that you describe the original scene from <Video 1> and you also describe the <Subject 1> from the <Picture 1>. In version 2 I split the different parts.
I’m running into a consistent issue when using a SAM3 (or similar) face mask together with a reference image.
The mask comes out at a different size/aspect and framing than the reference image. To make them match in the final generation I have to manually tweak the megapixels value on a “Scale Image to Total Pixels” node every single time I change the source image or video. There doesn’t seem to be a reliable automatic way to keep the face scale and framing consistent between the mask and the reference.
What I’ve tried:
Forcing the exact width/height from the Resolution Selector (both crop and pad)
Matching the mask size to the reference image
Using the same Scale Image to Total Pixels settings on both mask and reference
Various resize modes (crop / pad / longer dimension)
None of them give a stable match across different source images. The only thing that works is manually finding a specific megapixels value for each new mask (usually something low like 0.07–0.08 when the generation is set to 0.6 MP at 9:16).
It seems the model’s internal reference scaling (ref_image_size = match/max) combined with the variable framing of SAM masks makes automatic alignment very difficult.
Has anyone found a good automatic solution or a node that properly handles this mask-to-reference scale matching for MiniMax H3?
I don't have this problem, can you share your setup? I use the same size frames and masks to blend first and only after I resize them.
It's hard to tell what you mean here. There is only a mask between SAM and composite, after that it's just frames. Frames get inverted and sent to compositor, they also go to SAM to make the mask, which then uses it immediately to composite the originals with the inverted frames based on what you detected. As long as the source of the frames for both comes from the same loader, there cannot be a mismatch. But maybe that's not what you mean. My reference images are all square, 1024, 2048, 512, stuff like that. There is no issue from this though, I pull my W and H directly from a video info node, straight to the encoder. The composited video with the 'mask', which is not a mask at this point, goes straight to ref video. No scaling for me. I DID have issues there. So that might be a problem for you. Once I ditched source scaling, no problems. I also use <Picture 1> , not [Picture 1], that helped as well, mutatis mutandis.
The issue you describe was my exact experience trying ref model for head swaps. Ignores ref face, recreates video. This is clever. I had to do a lot of tweaking to get it to work with my face library, everything is cropped and masked faces, cropped square. Changed the sampling setup a bit, but the core concept is adaptable and once you figure out what H3 wants it works great. Really nice work, such a simple idea, I would not have thought to try this, and I've tried every trick in the book. Thanks for sharing. Pity ToS precludes showing exactly how well it works.
Thanks, if you get a better iteration of this solution let me know and I will update this. I hope I will be able to upload a new version soon.
share an promt of a face replacement . Do you crop your face into a square and then mask it?
@dav79mail156 [SUBJECT & IDENTITY]: Let <SUBJECT> be the woman with the exact facial identity seen in <Picture 1>, <Picture 2>, <Picture 3> and <Picture 4>.
Elsewhere in the prompt I just refer to <SUBJECT>. Once it has the reference, it knows what you mean. The more pictures, the better. Well, I'm talking about a real face here, not a fake one, so 'better' here means an actual likeness.
I've already build extensive libraries for faces going back years, so yeah mine are already cropped square. No masking needed though. Just standard angles. I run each through a resize though and feed them all at 1024. If they overpower the source video, can try smaller.
Really this model is super smart. Just be specific and tell it exactly what you're giving it. Usually you can figure out what's going wrong by looking at the output and fix it by clarifying.
The rest of the prompt just handles the source:
[SCENE & STYLE]: A video matching the exact camera angle, perspective, and atmospheric room lighting of <Video 1>.
[ACTION & TIMING]: <SUBJECT> performs all the continuous actions, head turns, natural expressions, and body movements seen in <Video 1>, maintaining strict facial consistency throughout the shot.
[BACKGROUND & ENVIRONMENT]: The background architecture, furniture, and environment remain identical and steady throughout the continuous take.
Stuff like that. I dunno, probably can be better. I've only been playing with it for a few days. Just getting crazy good results right away, especially with no ref video, t2v style. So it can only get better.
@Ponder_Stibbons thanks
PSA for the attention node you added. Install via git, it won't show up in comfy manager. This link is all you need https://github.com/Saganaki22/ComfyUI-sol-attn
Dude, your Sol-Attn node isn't showing up in the manager. And there are no links on your page. That's not good.
@lomote Can you tell me the difference between sol attn and sage attention? Thank you 🙌
Great Work.Could you create a version that's universally applicable to references? So that it includes audio, 8 images, etc., and allows the workflow to be used for other purposes as well?works great! thanks.
I'd recommend replacing your string boxes with the primitive multiline string box if you're trying to reduce custom node dependencies. That funcitonality is already native.
You should add the Sam 3.1 URL to this article:
https://huggingface.co/Comfy-Org/sam3.1/resolve/main/checkpoints/sam3.1_multiplex_fp16.safetensors
Nice. Thanks! I just created a small workflow that will generate copy/paste-ready inputs for the two scene inputs and the one character input your workflow requires, using Qwen3-VL. If anyone is interested, send me a message please.
It works amazingly well! I tried using the default example and the result is really good. The face perhaps is not perfect but that's a limitation of H3 (or perhaps just me using low resolution output)
Anyway thanks for providing this, it is a very neat technique.
How did you come out with the idea of inverting the person's image?
I noticed that the model would follow the inputs very literally if prompted to, so by inverting the colors the model must "decide" between following the input literally (and produce garbage) or produce a new video.
If you replace the subject with a black silhouette or a black rectangle it works as well, the model must "imagine" what is in the rectangle, but if you do that, you loss all the original information.
By inverting the colors, we keep the information (because it is there, just inverted) while we force the model to emit a new video.