Universal CLIP
You can use any universal CLIP, the "base" refers to the preferred base model
PONY_Base When Used in:
Pony - Full PONY character retention & SDXL name recognition (See Seed to Seed)
illustrious - Do not use PONY_Base-CLIP-L in illustrious
SDXL - Working Likely improves NSFW on models trained with that goal.
FLUX - Likely improves NSFW with minor loss on TEXT recognition
SD 1.5 - Not Working
SD 3 and SD 3.5 - untested
illustrious base:
May only work with the CLIP-G and in certain models.
illustrious - Paired with CLIP-G characters should prompt.
Pony - Working on Most PONY Characters but not FUTA
SDXL - Working Likely improves NSFW on models trained with that goal.
FLUX - Likely improves NSFW with minor loss on TEXT recognition
SD 1.5 - Not Working
SD 3 and SD 3.5 - untested
Single CLIP Loader
SD 1.5
Dual CLIP Loader
For FLUX use any T5 and either Universal CLIP-L (Pony_base has been tested more)
For SDXL you can use either universal base CLIP-L model with the Pony Universal CLIP-G
For PONY use Pony Base Universal CLIP-L and Pony Base G
For illustrious use Illustrious Universal and and Pony Base CLIP-G (Universal illustrious CLIP-G is still training)
Triple CLIP Loader
SD 3 & 3.5
Description
FAQ
Comments (37)
Hi there, what is this and how can it be used, will it work with flux base dev (fp8)? --- how NSFW can be improved? can you upload more examples?
CHeers and thanks
In platforms like Comfyui, you can use external CLIPs (checkpoints already has one built in. For sdxl, pony, illustrious and noobai, two built in). Those clips are generally 'default' and bundled inside, just like 'baked VAE' is. So people are using external clip loaders (in Comfyui you have, for example, DualClipLoader node) to improve those 'llama-like engines' to tell the AI better prompts in order to help with adherence (so more precise outputs). Other people goes ahead. They 'detach' the checkpoint , to become a unet/diffuser (a 'naked' checkpoint that uses fewer vram), then they load an external VAE and external CLIP (in the case of two clips, one being JoYPony-G or BigASP-G and the other, this one (Universal)). Beware that Illustrious checkpoints may not work with those clips because they were optimized for SDXL and PONY only.
It will not greatly improve base flux as the features your looking to extract with the clip are non existent. It has shown ability to improve feature extraction in models that have been trained on NSFW for FLUX
Felldude thats actually awesome, can you recommend me a great fp8 NSFW flux checkpoint? (there are a lot, i just dont know which to select)
ddflc27 Beyond whats popular I have no knowledge of any that are actual finetunes rather then Lora merges.
very interesting - now I simply have to test it ;)
That would be great
I'm really curious, are you able to put a few side-by-side comparisons with this CLIP-L and the default one to see the differences in the output? Thanks in advance!
I will write an article and explain the process.
Is this the same as in the JoyClip page ?
sorry for my ignorance, but make diference use an CLIP model apart? Aren't the output results much more dependent on the Unet in the model then CLIP
it is a genuine doubt, I just don't understand how it works yet, sorry and thanks.
In many cases the CLIP difference is minimal, however in custom models such as PONY and illustrious it is far from that
Ditto. I still don't know how these things go together, such as what is the difference between Clip-L and Clip-G or even the difference between Unet, Checkpoint, diffusion model, etc.
i've seen many CLIP models releases, but i never was able to understand how to install/use them on A1111, after reading all the links provided. it may be black magic instructions for those who can read between the lines perhaps... or comfyui users ?
CODOLOW Comfy is connect one node, super simple
Felldude thank you. but as said i use A1111... no way to use it in ?
CODOLOW For flux yes but last I checked not for sdxl
Take a look on my Post
https://civitai.com/posts/19741954
using a XL Checkpoint an loading on Forge the clips ... just like u change the Vae,
combinations in the image, top right
I tried them out a bit. PonyG+PonyL did not work on Illustrious at all for me. IllustriousL + PonyG on Illustrious can get weird camera angles and extra limbs sometimes. Depends on the check point tho. Im guessing when the CLIP-G for Illustrious comes out, it'd be a better fit. Looks good overall.
👍
Felldude I tried the IllustriousL + PonyG combination on some non-realistic Illustrious models and those did not work at all.
nickname45 The clip is far more aligned to realistic, but still that is interesting. I wonder if the 0-250 timestamp training I did with my anime model is why it works with no issue
Felldude to be precise, the 3 that I tried are: WAI-NSFW-illustrious-SDXL, Smooth Mix (Illustrious2 + NoobAI), Ri-mix - [PONY + Illustrious], not sure if that might provide any clues. Prompt was from your example image for IllustriousL CLIP.
nickname45 There might be some slight variation from v1 to v2 also...as if illustrious did not cause enough issue
Oh and I did see one mistake I made when testing I was using a trained version of G that is only released in my model you would need to pair it with that
https://civitai.com/models/1769561/illustrious-joy Has a custom version of G and might be the missing link
Felldude Your hunch was right, using the JoyClipG worked on all 3 models. SDXL LoRAs didn't work like when I tried PonyG + PonyL on a Pony checkpoint but I'm guessing this is expected since the Illus JoyCLIP G wasn't realigned in the same way. Tried a handful of characters and had no problems
nickname45 Cool, thanks, if nothing else I will upload that version of G here, if this training fails
Felldude the training did not fail lmao. Universal Illustrious G+L worked on Illus, Pony, and Noob for me, on both realistic and artistic checkpoints. Character knowledge seems to work fine, even for Ladybug. 1 CLIP to rule them all
nickname45 awesome
Hey, this looks great!
Do you explicitly recommend using this over JoyCLIP for Pony models now?
Not explicitly, it does have far greater laion alignment but certain models that are heavily finetuned on ponyv6 without sdxl merge in may do better with joy
Could be finally the bridge for merging XL and Illu Models on the clip thanks
Felldude OK thanks, noted! Anecdotally, in realistic Pony tunes it seems to be producing results that are at least as good as JoyCLIP, which is nice.
roe2 Good to hear,