For anyone with an ML background who want's to understand the grafting methodology.
This is a finetune-by-graft. Or maybe a GST - grafted shift of transformer (cross-architecture). I made both up, because there aren't any projects that have done it that I know, except one reddit post that made me look into it. I experimented with Wan and LTX on the side which led to the initial LTX Eros scripts that became what powered this, all before H3 ever came out. It seems like unified unbiased models like MMH3 can technically take attention influence from any other DiT without breaking if done correctly. Anima, Krea2, LTX, Wan2.2, Flux1 were all tried out, configs tested, about ~40 hours maybe of working in the dark without any paper or technical documents from Minimax. Eventually I developed linear-magnitude blend application and specific block and head gate targets allowing for a smoother graft on an attn-triplet-unfused version of H3 output as a patch file. That sent to lora extraction, then merged to checkpoint at taste. This is a merge but a merge of LoRas I extracted that interact to produce this current shift. I saved 5 ponds of water by recycling data in a few minutes on a single card instead of toasting a server up.
Turbo not recommended yet for i2v, especially when used with other LoRas. T2V use with turbo is better. Use 20-25 steps normal sampling with no dialogue, 25 steps with dialogue along with cache nodes and attn modes. More steps over 25 are not neccessarily better, and can be worse. Use full int8: int8 model, int8 VAE (if it doesn't crash comfy), int8 qwen3vl along with current cache or attn mode nodes. For smaller cards: quants, macOS ports, and Wan2gp support will likely appear on huggingface but not from me.
Known quirks:
Audio difference v.s. Base - This model's audio changes come from attention shifts seeking alternate audio pairing. Attn triplets were unfused before graft, both standard and triplet q_attn was grafted holding about maybe 10-15% audio influence, attn_k was frozen and MLP fc2 layers were untouched resulting in minimal audio interference. This was the main issue with the entire transformer graft and protecting audio. However this version is slightly louder overall than the base model.
Low resolution detail smearing - Some finger digits and fine motion will smear more at low resolution, also a problem in base model. As memory use gets more efficient increase resolution or work on the composition to get around it.
Odd outputs - This can attempt certain concepts more liberally than base model, but that can lead to some undesirable outputs in bad prompting and certain contexts. Data shift comes from completely different transformers and architecture. This shouldn't even work, so it is what it is.
This model is not dedicated to NSFW as that would violate community license agreement. Sure it can do it, just like base. Any NSFW generations are purely the result of advanced reasoning and tokenization resulting from experimental changes. All terms from the H3 community license also still apply to the users of this version. Don't be a dumbass.
H3 usage still requires very intense prompting for maximum effect. Every motion, every interaction, every sound plainly and fully described. Not with slang terms; with proper actionable words that can be tokenized. Refer to the h3 developer prompting guide, hand that .md file to an LLM or Chat agent and have them enhance or refine prompts along the released H3 developer prompt guide styles using the model's tag system. Certain concepts can be made from pure token reasoning. Consult the prompts in my previews to see certain physical descriptions that I use for some things. When using enhancement give the agent feedback about any issues in the generation and get them to describe motions in alternate fashion, or manually edit it yourself adding a negative like "no X, no Y". Still requires prompt refinement and trial/error for best outcomes.
Sulphur Project has 10k banked to attempt actual tuning. Right now training pipelines are sub-optimal. As always Eros is my personal side project, and this beta was also essentially a speed-run of finetuning, figuring out exactly in what configurations and target areas do you get helpful/harmful changes in the model. This is also a proof-of-concept of what and where to target while leaving the reinforcement quality of base unharmed by being additive.
Future Support:
Full bf16 - Doesn't seem necessary, train for this on the base model.
Ref model - In progress.
Workflows
Future versions focusing on further Audio and Motion improvements using actual gradient training, if I'm allowed to.
Description
Initial release-worthy grafted shift tune.
FAQ
Comments (43)
great!
Aahhh yeee!
Looks amazing! Cant wait to try.
This flv2a can still be used in reference workflow but only with image inputs. The ref model is still needed for actual full video and audio reference.
Good info! The reference mode is awesome, but the reference model needs more work.
Wow, can't wait to try it. Amazing work
wow h3eros version nice, h3max look good but so heavy i think, and some test say take more time to generate than ltx.. i think i will still use the ltx
My better LTX Eros gens with the extra sampling steps took about 1-5 minutes for really long ones. This model takes 1-5 minutes as well. All you have to do is keep resolution and length relatively low, there is a point where you start offloading too much because the latent is too big and it slows down.
plz ref2v
ypikayey
Can you make an A/B test or two vs minimax H3? Same prompt, same seed?
I find this fascinating but... I'd like a more objective illustration of what the graft actually does, if that makes sense.
I did probably 20 A/B scenarios that's how I even know I had a good configuration going. The differences aren't profound because it's the same model but they diverge more and more when you go in the direction of the donor model. It's like bread v.s. bread with some butter on it.
Can we see those
@Kingp0dd529t here's one but it's NSFW. The differences aren't substantial but they matter, and it's small details and motion fixes across pretty much everything. Until I find ways to keep pushing it further. The goal isn't to do much of anything to the base model besides fix it up. It's pretty good already and there's loras for anything else.
Appreciate the upload, that's interesting.
It's... very minimally altered vs the base model, at least in those examples. Which is good, I suppose.
@alphaatlas100844 Yes it is good. I've tried stronger weighting and it starts doing a bit too much. It's already maybe a bit too much Wan influence, if you notice the slower motions (Wan is 16 fps)
@tenstrip Have you noticed if this one is better at any concepts the original one couldn't do as well? I kind of understand a good bit of ai and training and blocks, seeds, etc... but I've never heard much about grafting. Very curious how to even go about grafting and if it can introduce or reinforce lost knowledge or if it's more of a detail fixer and/or follows prompts better.
@Light7799 It's something I guess I started, so yeah you haven't heard of it. Not sure what it is besides a training-less way to change the model's self attn. It can work on any model tbh, I tried H3 -> LTX 2.3 as well. Works way better when it's actually shared architecture of course, the last two versions of Eros were made like that.
Hi I already try your ltx 2.3 eros it work great on my system so will this also work with 64gb ram and rtx 5070ti ? it almost 40gb in size so i would assume it will not...
The int8 model variant is under it that's the one to use, and smaller than ltx2.3 actually but needs 25 sampling steps.
@tenstrip Thanks i will try it now
any recomendations for a good h3 workfkiw ref to vid
I've just started using the reference model today. I was focused on i2v and t2v so far. Both of the default comfy template workflows are pretty good for the model they just need the speed-up nodes put on them.
https://civitai.red/models/2663838/plaguekind-minimax-h3-ltx-workflow-ease-of-use-eros-or-sulphur-compatible-or-faceid?modelVersionId=3219462
try this, simple and it comes with upscaling method and memory chunk control
Dasiwa's fore sure
Very good model, my only problem is that each step takes much longer, 1st step starts with 180it/s, at 4th step at 300it/s, I can't figure out why, other models do 112it/s with the same setting, any idea why this is happening, am I missing something? Thanks!
High it/s like that isn't actually good you're bottlenecked at offloading using too much memory. The int8 on this one is the first one I got and it's 1.1g larger than the base pruned int8. I'd add the normal full row wise one but I don't have one, but you need to turn size slightly or turn down time by a second.
@tenstrip Yea I know I'm offloading, but I'll take it for the better resolution, also works good with Turbo LoRA, I lowered the resolution and don't have that issue anymore with your model. Didn't think this 1gb would cause that, thanks! Testing the model now
@suekapo What resolution do you use and what is the turbo lora?
Some one just told me to use FL2V model even when doing ref2v and it works great. No lora incompatibility, and the quality and speed of fl2v model!
what prompt u use to make the jump / cutscene / ?
@chaiwala008215 at 0:05.000 the camera cuts to [Shot 2]
I have to remake both grafted pieces from scratch using the ref model, so that's in-progress.
Is Gay stuff possible? Its often ends up in "hetero" sex.
I feel like the model follows prompts so literally that if you describe it enough with the right i2v start, then yeah it could work.
@tenstrip I am using text to video.
@CyclopsGER Yeah you'll need gay loras and a penis lora then.
fp8
Love your LTX model and excited to play and learn more about this one! Is there a place where I could look up some of the things this model was trained to do to streamline the learning and exploration process?
There was no training. This is altering the model's behavior more deeply underneath it's reinforcement with attention grafting on block heads. That's part of the goal/experiment since with a model this good the last thing you'd want to do is just SFT it on a couple thousand clips and negatively impact all seeds. I don't think a model like this without a proper training pipeline or an alternative version is trainable like older non-deterministic diffusion models were.
int 8 seems to run a lot slower than H3 int8 convrot pruned
Yeah I'm trying to get a normal int8, the skip edges runs mixed or something slowing it down.
I saw u upload ref2va version on hugging face. will u upload on civitai? cuz download from HF is so slow for me. it's great work! thanks :D