Introducing Proteus-RunDiffusion
In the development of Proteus-RunDiffusion, our team embarked on an exploratory project aimed at advancing the capabilities of AI in art creation. Our journey, inspired by the broad achievements of models like Pony Diffusion v6 XL CLIP, led us to experiment with the CLIP architecture in novel ways. Through a serendipitous process of trial, error, and discovery, we developed a unique approach to retraining CLIP that we hadn't initially set out to achieve. This approach inadvertently unlocked new potentials in character recognition, natural language processing, and, most notably, the versatility of artistic expression.
The cornerstone of our discovery, which we refer to as "style unlocking," emerged unexpectedly. This breakthrough allows models that were previously limited to specific genres or styles, such as anime, to generate art across a broader spectrum, including high-fidelity photorealism. This was a result of our reimagined CLIP model's ability to interpret and understand prompts in ways that surpass the original boundaries of style and genre.
We have observed that this retraining has also led to significant improvements in handling CFG scaling, effectively broadening the range from 3 to 50 without the previous limitations or failures. This enhancement opens up new avenues for creative expression and technical reliability in AI-generated art.
In terms of usage, we recommend a CLIP setting of -2 along with a strategic use of light negatives for optimizing the artistic output of Proteus-RunDiffusion. The CFG setting can vary depending on the project, with 8.5 being ideal for standard requests and 3.5 for more artistic explorations. The model supports and encourages experimentation with various tags, offering users the freedom to explore their creative visions in depth.
Using Proteus-RunDiffusion: Expect a Different Experience
When you start using Proteus-RunDiffusion, be ready for it to behave differently from other AI art models you've used. It's been designed in a unique way, which means it will respond to your prompts and commands in its own style. This difference is part of what makes it special, but it also means there's a learning curve. You'll need some time to get familiar with how it works and what it can do. So, as you begin, keep an open mind and be prepared to adjust your approach.
Importantly, we want to clarify that our development of Proteus-RunDiffusion was inspired by existing works but does not directly incorporate or rework specific components from models like Pony Diffusion's CLIP. Our advancements are the result of our proprietary research and development efforts, aimed at enhancing the creative possibilities and compatibility across different AI art generation platforms.
As we continue to refine Proteus-RunDiffusion and delve deeper into its capabilities, we are preparing to conduct a Human Preference Study and to share our findings and methodologies in more detail through upcoming research publications. This model represents not just a technical achievement, but a step towards understanding the broader potential of AI in the creative process, discovered through inspiration and the unexpected turns of research.
https://rundiffusion.com/proteus-rundiffusion#view-generation-samples
Description
since the prompting is a steep learning curve for a lot of people. expect easier prompting but not the same level of quality.
FAQ
Comments (24)
Are you guys planning on sharing any details on your discoveries regarding CLIP, so we may train better models as a whole?
It's really, really good, but PonyXL is better and famous BECAUSE of its nsfw power.
could this not be merged with Pony?
I don't really like all the weird tags needed with Pony (or I guess this one really)
@Janet PonyXL has a similar tagging system. Makes it substantially better. I extracted a Lora from this model and use it on top of a Pony merge. It makes extremely good images now. Give me a prompt in a dm if you wanna see.
its not much better than pony or playground.. but it seems trained on other images. So its a good model for some variance. I like when models are trained on unique new data ^_^
@yofoton174609 That's because you're using it wrong. Look at my horse picture.
@virtualfix6885 When you say you extracted this model as a lora and put it on a Pony merge, do you mean that you extracted this using SDXL 1.0 as a base? Or PonyDiffusion as a base?
@Jellai I believe SDXL. I also merged the Pony model with other stuff beforehand.
withclip is easily the best checkpoint for photorealism. The thing is most people are just going to dismiss this because they don't have a clue how to prompt with it and to be honest I haven't been able to get amazing results until today and I've got a lot to learn still. Unfortunately most people will need definite and clear instructions on how to use this or they will just dismiss it. A few, like myself, will do some research and get to use a fantastic, unique checkpoint. Keep up the innovative work!
Could you please share some tips for the newcomers ?
@fertyt449107 "score_9, real photo," followed by tags. You may want to test the tags on there own because if it doesn't understand them I usually get a picture of some sort of police officer (it's bizarre I know). it doesn't seem to understand any exat camera or film types. Does understand stuff like GoPro or polaroid. responds well to characters but if you put in something like 1960 it doesn't understand it. But if you put in a character from the 1960 or a show like Spock or star trek it'll give you the 60s vibe. It works very well with add detail lora and simple positive textual inversion in the prompt. It doesn't pick up a lot of stuff but what it does pick up it does very well. (illustration painting anime cartoon:2) In the negative. You can also use the pony diffusion photo Lora to get realistic images but it tends to lose versatility and just give portraits and not understand a lot. But definitely try it in an up close portrait. Also doesn't seem to work with a lot of Loras for whatever reason. It also works well with a short.prompt without tags like: score_9, real life, polaroid, the joker playing poker
@Jimsmithwick Thank you!
Well for practice i am gonna try the Neo thing with the bullets from the Matrix :D
@fertyt449107 ya see if it picks up the matrix or neo. It's very good with hands. Most of pictures have disappeared for some reason on here
which models are you comparing it to when you say "easily the best checkpoint for photorealism"? please share example photos with your prompts. and what research did you do? as far as I can tell the creator of this model has not published any details on their discoveries or ways to replicate their findings.
@Jimsmithwick This isn't a Pony based model in any way, what are you talking about lol
@diffusionfanatic1173 originally the authors of this model were claiming it was derived from the same training methodology as PonyXL. But this was just a marketing angle that they've since edited out. Overall it's still not a very good model and the creators get a bit shady/cagey whenever asked to back up the claims they make.
@ivorysky sorry haven't been on here in a couple of months. I'm taking back what I said about the clip version. When it gets your prompt it's excellent but it doesn't understand 90% of what you feed it. As for the pony tags I dunno, maybe it's like some placebo effect to be honest. The first few days I thought they were actually working but I dunno. Perhaps it's no different than adding a comma or removing a word to improve an image. However I do think the without clip version is the best model on here. I haven't used be clip version in two months as it doesn't understand very basic stuff.
great model，The images generated are impressive and I hope the author develops more and more powerful models!
Different between proteus 0.4 and this rundiffusion model?
I'm a noob and have no clue why but this model is interesting to me and i'm trying to figure it out. Thanks to the author(s) and team for making it. Hopefully i can figure it out!
very powerful model for real, it can do realistic things that no real model can in magical way ,holy crap man, you have to keep working on this model, it will put other model behind , cause it can create real images and add magical touch on it without losing the realistic part from it< like damn high quality movies and it do PERFECT JOB
i just have one problem , the model have hard time to create two people together
there's
1- rundisffusion-xl
2- and Proteus-RunDiffusion-xl with clip
3- and Proteus-RunDiffusion-xl without clip
4- and Proteus-RunDiffusion-xl dpo truereverse
5- and Proteus-RunDiffusion-xl dpo reversesmooth
..help me please because i'm lost.
what model should i download and use , if in simple compare tell me what is more better than other in which usecase?
This sucks. Every image is terrible.