Endless MiniMax H3 (with Endless LipSync) v1.0
When MiniMax H3 came out, I was badly missing the Endless Wan 2.2 I2V (SVI 2 Pro) features, so at first I look at other options, but most of them included an AIO node that could do everything, and I couldn't use most of my workflow, because they did everything inside that huge node. The only exception to this, was ComfyUI-H3-Motion-Context, which I could easily integrate into my setup. Its big shortcoming though, was that it just created single videos. No way to concatenate them without the lossy step of decoding and re-encoding in a video editor. So, I created a custom node that could do just that, and voila..
Endless MiniMax H3 (with Endless LipSync) v1.0 A simple workflow to create MiniMax H3 videos of unlimited duration, using ComfyUI-H3-Motion-Context and H3 Motion Context Clip Stitcher. I can easily create a 1:30 lip-synced video myself, with a RTX 3060 12GB at around 2 hours (with retries).
It can use both FL2AV and Ref2AV, and can also create normal MiniMax H3 videos. The extra parts though, are the saving/loading of the latent from every generation we do, and the stitching of all (or some) of them, whenever we want a full video. Nothing visible at the connections, no indication that there were more than one video. We can try and re-try every generation, looking for the best one, and then proceed to the next. We can re-do any previous generation if we like too, but we will not be able to use the clips after it (previous clips are not affected), because they are in a way, "fused" with the replaced one.
Controls
The RED nodes Enable/Disable parts of the workflow.
Generation Mode (select only one)
T2V / I2V (FL2VA)
REF2VA
Optimizations (select as many as you want, but only 1 Attention and/or only 1 Cache) This panel controls the nodes that are inside the
Optimizations/LoRAsubgraph.Reference Items (select as many as you want) Special usage for the
Audio 1 Forced/Multi, more for them later.Setup (select as many as you want depending on the goal)
Generate starts a generation. You don't need this if you just stitching clips
Preview enables the main video preview that can show you where the generation is going before it finishes, so you can stop bad generations
Use previous clip, uses the last part of the previous generation to start the current one, continuing the video. Enable it if you want the current generation to be stitched with the previous video
Stitch clips, stitches all the clips (depending on the Stitcher's settings) from the
h3_contextfolder (this is the default folder that the H3 Motion Context node uses to save the latents)Stitch last only, stitches only the last video with the current
Save Video just saves the generated video
The GREEN nodes are various settings nodes that must be setup. Apart from the Prompt, Seed and LoRA nodes, the most important are these:
Configuration It contains all the settings for the video. The most important setting for the clip stitching, is the Save to clip Index number. This specifies the clip file's number, that the latent of the current generation will be saved to (overwriting any previous existing file). This number also tells us who is the previous clip file that we use to start our current video clip.
Setup multiple Forced Audio clips This is used if we use our own audio for the video, and we want to also stitch many clips together. More about it in the Usage section.
Usage
Most of the settings are self explanatory (like Optimizations or enabling Reference items). Here, I will just list the main goals of the workflow
Create a normal video
Select Generation Mode
Enable Generate, Preview and Save Video
Save to clip Index to 1
You get a video and that's it.
Create a lip-synced video
Select Generation Mode
Enable Generate, Preview and Save Video
Enable <Audio 1> and load an audio file
Enable <Audio 1> Forced
You get a video that is lip-synced with the provided audio
Create an Endless video
Create a normal video
Enable Use previous clip
Save to clip Index to 2 and generate
Save to clip Index to 3 and generate
Save to clip Index to 4 and generate
...
You get many small videos, each one of them starts with the ending of the previous one. At a later time, you will use the Stitcher, to stitch them all together to one full video.
Create an Endless video, stitched
Enable Stitch clips With every generation, the Stitcher will stitch all the previous clips with the currently generated one. This way you always get the full video to check.
If you also enable Stitch last only, only the previous and the current videos are stitched together, so you can check the connection without waiting for the full video to be created. You can always stitch them all together at the end.
You get a full video every time, or just the last 2 videos connected, for previewing the connection.
Create an Endless lip-synced video, stitched
Create an Endless video
Enable <Audio 1> Forced and <Audio 1> Forced Multi
At the Setup multiple Forced Audio clips panel there are some settings.
Start offset: At the 1st gen, you put here the initial offset that you want for the song (e.g. where the lyrics start). After every successful generation (when you advance the Save to clip Index number), you must copy here the value that is in the
Copy to Next Start offsetbox.Frame offset (ignore if 1st clip): Never mind at 1st generation. After every successful generation (when you advance the Save to clip Index number), you must copy here the value that is in the
Copy to Next Frame offsetbox.context_length must be the same value everywhere (here, at the
Motion Context, and at theH3 Motion Context Clip Stitcher). It's the number of common frames the 2 video clips use to blend together.
You get a full lip-synced video every time, or just the last 2 videos connected, depending on the Stitch clips and Stitch last only settings.
Just stitch the clips together You just have to enable the Stitch clips and the Save Video All (or some of them depending on the settings), of the clips in the
h3_contextfolder, will be concatenated to a single full video.
Notes:
All generated clips that need to be stitched, must have the same dimensions.
You can organize past generations in folders inside the
h3_contextfolder, since all the nodes look only in the root of this folder for clips.context_length must have the same value everywhere: at the
Motion Context, at theH3 Motion Context Clip Stitcherand at theSetup multiple Forced Audio clips(if you are using it). It can can have only the values of 5, 22, 39, and 56.You can stitch together clips that are generated from either fl2va or ref2va.
In this workflow, I don't use the normal ref2va model in the
Reference to Videonode, but rather the fl2va with theref_lora_layer20-49adalnLoRA that has better quality. You can check some LoRAs with different weights here, or totally bypass the LoRA and use the normal ref2va model.
Models used:
Custom Nodes used:
Description
FAQ
Comments (42)
Hi, thank you for sharing. I was able to creat first T2V clip, but when I try second T2V with Use previous clip = Yes, I get an error:
File "/home/apps/ComfyUI/custom_nodes/minimax-h3-hybrid-cond/model_base_patch.py", line 30, i
n extra_conds_with_hybrid
visual_latents.extend(item["latent"] for item in keyframes)
File "/home/apps/ComfyUI/custom_nodes/minimax-h3-hybrid-cond/model_base_patch.py", line 30, i
n <genexpr>
visual_latents.extend(item["latent"] for item in keyframes)
~~~~^^^^^^^^^^
Can T2V be used in you WF to continue from previous latent ?
No reason not to. You should mix and match between T2V, I2v and R2v..
Haven't tried T2V, but it should work.
Read the instructions and be careful with the Save to clip Index number. Start with 1 and then change it to 2, 3, ..
Also, minimax-h3-hybrid-cond is not a node of the workflow, so, something is wrong here..
@noembryo
Thank you, the error comes from https://github.com/kitsune123150/minimax-h3-hybrid-cond which creates a model patch regardless if used or not. The patch was borked and needed a fix.
Continuation now works.
God bless you, I CANNOT wait to try this - Does it have identity drift? like sharpness / saturation increase? I've been trying talking head videos but results always fall apart like 4 10 second videos in
I didn't notice anything like this, but I don't really know what you asked the model to do.
Perhaps, identity drift (if exists), can be avoided with the use of ref images?
Try it and tell me about your findings..
@noembryo So the problem I noticed with chaining videos along is that it just gains progressively worse sharpning,saturation & other gradual increases over time, I've tried for 2 weeks to get a talking head video to work for about a minute without wild changes
I used proper terminology, (subject / picture)
reference images with grey bankgrounds as well as a scene for them to be located in
This was me using context loop, seed hunter, and a handful of other workflows, but I'm about to give this one a go
I think it might just be a limitation as you're generally feeding it the last frames of a previous video and it just slowly changes over time, but I'll report back on this one
@datlurkaa So, you already checked the constant reference of a specific image at every prompt?
Remember that it doesn't work for FL2VA, because the 1st frame is ignored due to previous video usage..
@noembryo Yea I've been using REF2V w/ the reference for each continuation - not FL2VA,
For your WF I'm following instructions for REF2V which is attaching the ref, using the clip index to 1 for the first video, generating the first video, then adjusting it to stitch previous video whilst leaving the reference attached correct?
@datlurkaa Yes, (but change the index to 2 ;o)
Also, if that is not enough, you could also reference the image in the prompt..
I tested it with 3 separate 10-second clips using the REF2VA path in the WF.
For every clip I kept the exact same reference image attached as Picture 1.
Clip 1 was generated normally.
Clip 2 used “Use previous clip” with index 2.
Clip 3 used “Use previous clip” with index 3.
I kept the same resolution/settings and only changed the dialogue prompt for each clip.
I changed the config dial to use previous clip & stitch lips
Unfortunately I still saw progressive quality loss. Clip 2 was slightly worse than Clip 1, and Clip 3 was noticeably worse again - mainly increased sharpness/processed look and some overall image degradation.
So even with the same reference image being supplied on every REF2VA generation, the quality still seems to accumulate/drift as the chain gets longer.
This could be because there's no camera cuts or anything like that, maybe I'm doing this wrong.
Do you mind testing out a single scene, no cuts talking video thats chained, I'm really curious if I'm just being dumb or its just a limitation of how these work
@noembryo I'm using ref images:
I'll recap what I'm doing:
Reference image enabled (1)
WIthin my prompt I reference subject 1 as image 1, retain details, etc
I have dialogue for subject 1 where they speak - Each clip I only change dialogue, everything else remains the same - image ref still stays in slot 1
For settings, everything is on default
I generate a video as clip index one
I change clip index to 2
I change dialogue
Generate the next video
And so on and so forth,
But each video increasingly worsens the detail of the scene
I am using the vanilla WF you have uploaded here, i didnt really change anything else - using all the same nodes & models as they are.
@datlurkaa Well, I created a 52 sec video of a talking head using an audio file, and unfortunately the results are bad. There is an increase in contrast with every new segment, so by the 6th clip the contrast was truly too much.
So, it must be a problem with ComfyUI-H3-Motion-Context that apparently all the workflows share.
I opened a GitHub issue.
Can you tell me if the other workflows that you used were based on ComfyUI-H3-Motion-Context too? I know that context loop and seed hunter do, but is this true for the others too?
@noembryo Glad i'm just not being a dumbass lmao, yea so the workflows I tried was Multi-Shot, Seedhunter, ContextLoop, Muse Director - Not all of them used Motion context (I think) You can double check but this issue has been driving me insane, I haven't seen any workflow crack it yet fully, The best performing one so far has been context loop
@noembryo I think the only one that doesn't use it that I've tried is multi-shot but I think it suffered the same fate
@datlurkaa As the creator of Motion Context told me in the issue is:
"This has nothing to do with Motion Context. It is an unfortunate result of changing clips with any video model. Wan, LTX, H3, doesn’t matter. The process of sampling the latent frames results in a small percentage of contrast bloom, detail loss, and audio dampening per chain. The community as a whole is still trying to figure out how to combat this."
@noembryo Ah, I was hoping this wasn't the case - just a limitation in general - Thanks man
After multiple testing, aswell after 4th gen, quality degrades. Nonetheless, great worfklow!
@obamarx 🙏
Outstanding work, you nailed it! If I could recommend, "path" load is not allowing simple drag & drop, but aside from that, GG.
Thank you. Do you mean the "path" load of the "Load image (from path)" node?
@noembryo Sorry for the late reply. I've been doing a lot of testing. Your workflow is top notch, it's only problems related to ComfyUi (which are easy to fix for my preferences). That said, the model is hard to work with! Thus, with your demo, you are acheiving much better results that I could (I've asked Fable 5.1 for help), but with the community help it's still work in progress! Just like LTX 2.3 vs LTX 2.5, 2.3 is still better for lip synch compared to the latest. Long story short, thank you for sharing you approach!
I think they are saying, one is unable to drag and drop to load image nodes.
@voxefx788 Unfortunately, drag and drop is restricted in ComfyUI (inside a browser).
It only allows you to copy the dropped file to the standard "input" folder, which is something I really don't like (it's the reason I made these "Load image (from path)" nodes at the first place)..
Thanks for sharing, man. I tried 4 5-second clips. It worked fine till the 2nd one but the 3rd one had shimmering / crawling halo around the entire silhouette of the character and throughout. Got worse in the 4th clip. I am using Larry's 4-step lora so I did replace the lora node and Get sampler node with the ones that's required for this Turbo lora to run. I've not tried this H3 motion context node before, maybe i'll add it to my my own MM workflow and check if it's the issue with the node or some config at my end.
Also the same frontend issue from Endless Wan workflow came back, If I refresh the page or even save and open it again, It'd have gone back to the original settings of your workflow. This is a different Comfy Install from the last time and is currently on the latest version. But yeah, this isn't my main issue at this time. I'll let you know what I find. Thanks again.
I can't say anything about your first issue, since I can't replicate it here (the demo video has 11 clips of ~12 sec), but for your second issue I know the answer..
ComfyUI frontend bugs.
After every ComfyUI update, the first thing to check is to refresh any workflow with subgraphs and check if it resets the values.
The last frontend version, that the subgraphs were working OK for me, was 1.39.2. To install this version, you must do pip install comfyui-frontend-package==1.39.2 in your ..\venv\Scripts\ folder. After that you will see a warning once, but other than that, everything will work fine..
@noembryo Thanks, man. I was able to finally able to get the workflow to work after trying a different model without a Turbo Lora. Worked perfect and seamless. Few questions and thoughts:
1. I generated 5 clips of 15s each, final clip of 1m:11s. The quality was the best in the first clip but there is a noticeable change as it goes to the next ones. The motion and prompt following remained good but the character's skin started to become more and more polished/baked. It does look like a logical side effect but I wanted to check with you if it's indeed normal and if there are any ways to maintain better consistency.
2. Your example music video, there are no quality issues there at all, not much anyway. Could you please share your secret. What prompt/s did you use:
a. to maintain consistency, b. for audio sync c. and for that smooth fade in/fade out transitions to different environments (that was so so cool)
3. You have already included Easy Cache and Spectrum among the optimizations, if more people ask for it, please consider adding SLA attention to the workflow as well.
Many thanks for your help on this.
@Pitiful_Season42942867 1. The degradation of contrast/details is bad, but unfortunately not yet solvable. We already discussed this here at the comments. At first I thought it was Motion Context's fault, but as its creator told me in an issue I opened..:
"This has nothing to do with Motion Context. It is an unfortunate result of chaining clips with any video model. Wan, LTX, H3, doesn’t matter. The process of sampling the latent frames results in a small percentage of contrast bloom, detail loss, and audio dampening per chain. The community as a whole is still trying to figure out how to combat this."
2. I think that the reason my video has less degradation, is the constant changing of scenes.
My worst tests were with a talking head, with no change in the frame at all.
The uploaded video was made in a hurry, just to have a demo for the wf, and the prompts were dead simple. They started the same way, and then I changed only the thing I wanted to see. Like "She gracefully sings a song, while the environment is changing around her. At 3 seconds, the background is transformed into a beautiful sunny beach, under a steep rock." and "She gracefully sings a song, while the environment is changing around her. The camera gets higher like a drone camera pointing down to her that looks up."
The audio sync was made using the "<Audio 1> Forced Multi" option of the wf (you can read about it at the WORKFLOW INFORMATION node).
3. I did tried that too, but I wasn't able to make it work/install on my system (Windows 10).
Maybe if I find a way to install it, I could add it to a next version. I don't believe is much better than ComfyKitchen though..
if you can provide tutorial video would be great.
I tried things and gladly it worked, but i am not sure how much we can join or how long we can create a clip.
I am able to create 15 sec clip with a normal workflow with 0.7 MP
With this my first clip generates with 0.6 something with 10sec max and the next gen always create problems.
I'm on linux server, rented GPU kinda thing so I cant see the videos inside. I had tried many things but very small like, 2,3 seconds clips can be generated which can attached and there are several issues which .. are over my head.
The most common error is
```
[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 19995MB Staged. 208 patches attached. Force pre-loaded 210 weights: 1175 KB.
12%|█▎ | 1/8 [00:15<01:49, 15.58s/it, Model Initialization complete! ]
[ERROR] !!! Exception during processing !!! FinalLayer.forward() missing 3 required positional arguments: 'sigma', 'sample_sigmas', and 'shifts'
[ERROR] Traceback (most recent call last):
```
in short.. may be video is better
I have disabled the stitching and it seems to work with 0.7 x 10s .. seems like i can live with that :D
Loved the continution
Unfortunately, I can't create a video tutorial currently, but I might be able to help you if I understand your problem.
First of all, remove the 2 nodes ("Load Audio (Path)🎥🅥🅗🅢" and "PlaySoundKJ") that are under the "Video Combine 🎥🅥🅗🅢" node, at the "06. Save Video" Group. These are not needed in Linux. They shouldn't create any problems but just to be sure.
The documentation in the "WORKFLOW INFORMATION" describes all the steps to create different kinds of videos.
So, starting with a freshly opened workflow, and except for the changing of the model paths to work in your system, what other settings did you changed for the 1st generation?
@noembryo Yes I have already removed them. The stitching is he problem but i am no longer stitching them in comfy any more. So far going good. Still need to understand what those FORCE audio videos are fore.
Overall good stuff, thanks
@PinkPromptPrincess The <Audio 1> Forced uses the provided <Audio 1> audio as a guide for the generation, and synchronize the video with it.
<Audio 1> Forced Multi does the same thing but for multi/stitched clips..
You can read about them at the "WORKFLOW INFORMATION" node..
I have isolated
> missing 3 required positional arguments: 'sigma', 'sample_sigmas', and 'shifts'
issue, might be on my end. It is Cache Spectrum, If I use EasyCache it works.
Good stuff liked it so far. Added this comment for others.
Cheers bro
Hey, I'm having way too much fun with your workflow, ahah. Works like a charm.
Before I forget, there's an issue in the workflow: You have 5 nodes to load ref images, but only 4 are connected (there is no Get_ref Image 5 connected to MiniMax H3 Reference to Video)
Also, since the generated clip need to be the same dimensions, I automatically create/use folder; linked with the video path AND motion context paths; named with the dimensions to avoid any mistakes on my part.
(Well... I actually create/use a [project_name]_[dimensions] directory containing the generated videos and h3_context folder, that way everything stay organized with minimal interactions, especially when I create prompts for several clip in low quality THEN regenerate everything with higher megapixels).
It could be worth adding to the official workflow ;)
I can upload a modified version if you're curious (it's really just a simple tweak, but it's sooo useful :P)
Anyway, keep up the good work!
Thanks. Yes, I would like to check it if you can uploaded it somewhere (for the next version)
Also, if you're interested, my updated custom nodes (noEmbryo) have a new node that can delete previous clips..
@Iuot Also, what do you mean by "regenerate everything with higher megapixels"?
Upscale the latents? If yes, how do you use the saved latents to do that?
I couldn't find a way to feed them to LTXVSeparateAVLatent, so I created another node (H3 Context Latent Converter) to do just that (included at my nodes)..
@noembryo Ok, I tidied up everything into a subgraph, added it to a clean workflow (and added the GET for ref image 5 while I was at it).
Uploaded it on pastebin: https://pastebin.com/ztTjcFL2
Basically, the subgraph:
-Take a name as an input (It work without a name, but it'll just dump everything in the base folder)
-Format the texts (It use only base ComfyUI nodes and one comfuyui-easy-use, so no new custom node. I also added the "Clip Index" in the video filename to know in which step it was generated)
-Output a video filename (for Video Combine), a context folder name (for Context Clip Stitcher and the Motion Context subgraph) and a context filename (for Context Save)
And no, not upscaling the latents, I mean changing the resolution:
I make my tests with a resolution of 0.2 megapixels (let say 608 x 352), make the prompt for the videos 1, 2, 3 and 4, and once everything is to my liking, I rerun the prompts of every videos from 1 to 4 at a higher resolution.
But if I did that without cleaning h3_context, I was bound to make mistakes and get errors when I forgot to backup/move things. Or if I wanted to switch back and forth between my tests and the "high resolutions" one, or if I wanted to generate something else entirely, I had to handle the h3_context manually...
Here, I just need to change the project name to create/continue something, changing the resolution is handled without opening h3_context, and I can dump all my test by deleting the resolution's folder. ^^
@Iuot Thank you. I'll look at it when I got some time.
BTW, by upscaling the latents you got the same exact scene only in higher resolution, and faster (since most of the calculations are already done).
Just changing the resolution, changes a lot of stuff in the video, even if you keep the same settings.
My only problem until now, is that I can't find a latent upscaler that gives a really good result..
@noembryo Ok! I'll have to take a look at it then
@Iuot Checked your wf, and I think it is really helpful. Nice work..