Todo: Teaching JUST CLIP_L how to behave.

CLIP_L, is an idiot. You can easily test it's response to tags and image input using the associated huggingface articles. It's essentially GPT2 trying to talk to T5.

https://huggingface.co/docs/transformers/model_doc/clip

So... we have an idiot in our midst. How do we ensure this idiot, doesn't say the world is made of potatoes?

Well there's a few options based on that, but since the T5 is essentially it's training buddy when it comes to the FLUX unet, we can't just throw it in the meat grinder and switch to CLIP_G now can we. We also can't just grab the pony version either, since the damn outputs are so different.

HOW can we TRAIN the CLIP_L itself, to behave in a way that the AI model understands, faster than training the UNET!?

Simply put, positive reinforcement training.

I as a person, respond considerably better to positive reinforcement, than negative reinforcement. If you attack me, assault me, directly berate me, etc; I often lash out, and even disregard useful information. It's in human nature I think, and I'm very human. However, if you give me kind words, cookies, etc when I perform the intended tasks; that is a faster learning method than pain.

Pain works, but positive outcome produces positive pathology. This will be our training policy.

We give it positive reinforcement when it's correct, and no reinforcement when it's incorrect. We cannot punish the AI, as it would destroy linkages and not help our process. We'll punish this AI, by not giving it cookies.

I dub this, directed learning, as it's going to be important when saving time on lesser hardware.

Images where the outcome is above a certain threshold, will simply be omitted from the training cycle, allowing the process to speed up.

Honestly I'm shocked that none of the commonly used training tools do this yet.

Kohya just keeps flooding images and prompts in, even when the image and prompt is completely learned, causing it to rely on the individual to determine the outcome, mostly due to the issue with identification and time.

HOWEVER, we can get a good idea what CLIP_L dumps out, because it's a very quick zero-shot system. It'll be easy to determine if certain results are in fact trained, by using validation images, with almost no time in comparison to it's larger scale counterparts.

I have a process, a program, a strategy, and the images. It's just a matter of training the CLIP_L that isn't an idiot.

As it stands, I can use my current CLIP_L as a kind of basis for comparison, since it's already beaten the majority of anatomical problems that the base CLIP_L will ignore or disconnect when trying to generate images. So it's just a matter of testing and implementation.

My CLIP_L is in fact an idiot still though, which is why it needs to be improved in a way that can work with the diverged flux UNET in the territory I'm headed. I can't go without a proper travelling companion after all.