Various ControlNet-RefLatents experiments for Anima Base v1.
Inference:
https://github.com/Mirumo0u0/ComfyUI-Cosmos-Reference/tree/main
https://github.com/Haoming02/sd-webui-forge-classic/commit/b99ba0d6c3f67acd6a16edac7eaa6cd721ce7deb
Note: For NeoForge "[Anima] Enable Reference" must be enabled.
Training:
Description
FAQ
Comments (11)
How does RefLatents ControlNet differ from ControlNet LLLite?
Bit of a complicated topic and I'm not really well informed enough on the underlying tech but it's my understanding that:
- ControlNet LLLite is intentionally limited in its architecture
- RefLatents targets a far richer and more comprehensive set of layers
This would allow RefLatents to not only operate on things like positional maps (Canny/Lineart, etc.) but conceptual operations (Edit, for example). Whereas ControlNet LLLite is highly unsuitable for anything except very simple, lightweight inputs.
I'll be exploring the technology more and will be able to speak more confidently on it soon.
EDIT:
So far, I've found RefLatents to be more strict (which is good), more reliable in tougher scenarios and more lacking in control (no strength slider, no start/end step).
It's my understanding that RefLatents does support start/end step, but so far the respective implementations simply haven't added this feature.
@inpaint patch for the Mirumo0u0 implementation: https://gist.github.com/cavemanextreme/0b53ebfd88fbb3e6124fc56cd995c50d
@cavemanextreme Very nice, thanks. I feel like RefLatents benefits from ending earlier to avoid the heavy effect on the result, allowing later steps to clean up a mostly formed image. I'd like to see this as a first-class feature (for example, ControlNet slider on NeoForge). Have you tested this and verified it does not degrade the result, likewise have you considered a NeoForge port? Both of these would potentially bring the feature into mainstream use.
@inpaint It may degrade the result when using kind of in between values, where the model tries to fix whatever the lineart is but does not have enough timesteps to do so, otherwise no.
I don't use NeoForge but any LLM should be able to create something like this if you point to the code you're using.
Could you make an openpose controlnet for Anima 2b base v1?
Getting error message at Ksampler
RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 152 but got size 150 for tensor number 1 in the list.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "execution.py", line 545, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
File "execution.py", line 344, in get_output_data
return_values = await asyncmap_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
File "execution.py", line 318, in asyncmap_node_over_list
await process_inputs(input_dict, i)
File "execution.py", line 306, in process_inputs
result = f(**inputs)
File "nodes.py", line 1607, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
File "nodes.py", line 1571, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
File "comfy/sample.py", line 74, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "custom_nodes/ComfyUI-TiledDiffusion/utils.py", line 51, in KSampler_sample
return orig_fn(*args, **kwargs)
File "comfy/samplers.py", line 1461, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "comfy/samplers.py", line 1351, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "comfy/samplers.py", line 1333, in sample
output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/samplers.py", line 1260, in outer_sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
File "comfy/samplers.py", line 1235, in inner_sample
samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "custom_nodes/ComfyUI-TiledDiffusion/utils.py", line 34, in KSAMPLER_sample
return orig_fn(*args, **kwargs)
File "comfy/samplers.py", line 1005, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
File "comfy/k_diffusion/sampling.py", line 218, in sample_euler_ancestral
return sample_euler_ancestral_RF(model, x, sigmas, extra_args, callback, disable, eta, s_noise, noise_sampler)
File "torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
File "comfy/k_diffusion/sampling.py", line 248, in sample_euler_ancestral_RF
denoised = model(x, sigmas[i] s_in, *extra_args)
File "comfy/samplers.py", line 640, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "comfy/samplers.py", line 1208, in call
return self.outer_predict_noise(*args, **kwargs)
File "comfy/samplers.py", line 1215, in outer_predict_noise
).execute(x, timestep, model_options, seed)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/samplers.py", line 1218, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "comfy/samplers.py", line 620, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "comfy/samplers.py", line 211, in calc_cond_batch
return calccond_batch_outer(model, conds, x_in, timestep, model_options)
File "comfy/samplers.py", line 219, in calccond_batch_outer
return executor.execute(model, conds, x_in, timestep, model_options)
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/samplers.py", line 333, in calccond_batch
output = model_options['model_function_wrapper'](model.apply_model, {"input": input_x, "timestep": timestep_, "c": c, "cond_or_uncond": cond_or_uncond}).chunk(batch_chunks)
File "custom_nodes/comfyui-spectrum-sdxl/src/spectrum_node.py", line 108, in spectrum_unet_wrapper
raw_real = model_function(x_real, timestep_real, **c_real)
File "comfy/model_base.py", line 196, in apply_model
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
File "comfy/patcher_extension.py", line 113, in execute
return self.original(*args, **kwargs)
File "comfy/model_base.py", line 240, in applymodel
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds)
File "torch/nn/modules/module.py", line 1775, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "torch/nn/modules/module.py", line 1786, in callimpl
return forward_call(*args, **kwargs)
File "comfy/ldm/anima/model.py", line 214, in forward
return super().forward(x, timesteps, context, **kwargs)
File "comfy/ldm/cosmos/predict2.py", line 847, in forward
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
File "comfy/patcher_extension.py", line 114, in execute
return self.wrappers[self.idx](self, args, *kwargs)
File "custom_nodes/ComfyUI-Cosmos-Reference/cosmos_temporal_reference.py", line 93, in cosmos_diffusion_reference_wrapper
x = torch.cat([x, ref.to(dtype=x.dtype, device=x.device)], dim=2)
RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 152 but got size 150 for tensor number 1 in the list.
Obviously I don't have your workflow so I cannot readily reproduce the issue. Have you ensured that the gen resolution matches your ref resolution? That's all that comes to mind for me. Otherwise please raise an issue on the Github repo directly to the node author as I am not able to help with this sort of thing.
@inpaint Okay, that's a really bad limitation when it comes to matching the exact reference resolution. I tested it, and it did apply some style transfer from my style LoRA to the reference image, but the character itself didn't change. There were also some artifacts because the prompt and the reference image seemed to be fighting each other over what should be generated.
@ikiru99percent Broadly, most ControlNet implementations have a resolution constraint, often the Node/WebUI/etc. will simply do this without informing you (resize/crop).
@inpaint Ok, Are you working on the new version for canny controlnet?