Introducing Proteus-RunDiffusion
In the development of Proteus-RunDiffusion, our team embarked on an exploratory project aimed at advancing the capabilities of AI in art creation. Our journey, inspired by the broad achievements of models like Pony Diffusion v6 XL CLIP, led us to experiment with the CLIP architecture in novel ways. Through a serendipitous process of trial, error, and discovery, we developed a unique approach to retraining CLIP that we hadn't initially set out to achieve. This approach inadvertently unlocked new potentials in character recognition, natural language processing, and, most notably, the versatility of artistic expression.
The cornerstone of our discovery, which we refer to as "style unlocking," emerged unexpectedly. This breakthrough allows models that were previously limited to specific genres or styles, such as anime, to generate art across a broader spectrum, including high-fidelity photorealism. This was a result of our reimagined CLIP model's ability to interpret and understand prompts in ways that surpass the original boundaries of style and genre.
We have observed that this retraining has also led to significant improvements in handling CFG scaling, effectively broadening the range from 3 to 50 without the previous limitations or failures. This enhancement opens up new avenues for creative expression and technical reliability in AI-generated art.
In terms of usage, we recommend a CLIP setting of -2 along with a strategic use of light negatives for optimizing the artistic output of Proteus-RunDiffusion. The CFG setting can vary depending on the project, with 8.5 being ideal for standard requests and 3.5 for more artistic explorations. The model supports and encourages experimentation with various tags, offering users the freedom to explore their creative visions in depth.
Using Proteus-RunDiffusion: Expect a Different Experience
When you start using Proteus-RunDiffusion, be ready for it to behave differently from other AI art models you've used. It's been designed in a unique way, which means it will respond to your prompts and commands in its own style. This difference is part of what makes it special, but it also means there's a learning curve. You'll need some time to get familiar with how it works and what it can do. So, as you begin, keep an open mind and be prepared to adjust your approach.
Importantly, we want to clarify that our development of Proteus-RunDiffusion was inspired by existing works but does not directly incorporate or rework specific components from models like Pony Diffusion's CLIP. Our advancements are the result of our proprietary research and development efforts, aimed at enhancing the creative possibilities and compatibility across different AI art generation platforms.
As we continue to refine Proteus-RunDiffusion and delve deeper into its capabilities, we are preparing to conduct a Human Preference Study and to share our findings and methodologies in more detail through upcoming research publications. This model represents not just a technical achievement, but a step towards understanding the broader potential of AI in the creative process, discovered through inspiration and the unexpected turns of research.
https://rundiffusion.com/proteus-rundiffusion#view-generation-samples
Description
FAQ
Comments (35)
(edited: below refers only to the original withclip version, the new withoutclip version addresses all of these concerns!)
The results are not good.
- won't draw celebrities, seems trained to disfigure faces if you mention them in any in any complex scenario (vs still maintaining a nice image of a similar person on ponyXL)
- struggles with prompt coherence relative to juggernaut, leosam, ponyxl, animagine, vibrant horizon ect... it's actually a bit worse than sdxl base model on prompt understanding.
- can't draw a penis even with a lora (it's maybe the most censored model on civitai on this front)
this may be an ok model if you need an extremely sfw model that doesn't require long prompts. But there are so many great xl models out there now and this one doesn't seem to improve on them.
This can do celebrities. It can do artistic NSFW. It isn’t selling anything we open sourced the weights and are about to release a paper on the methods used. < this claim very much confuses me. This is a prototype of a customer clip training model approach. We’re trying to further the technology.
if you feel this is a inappropriate place for this model to be published we are more than willing to either rephrase the description completely or close source the model.
also on a personal note come on man "But there are so many great xl models out there now and this one doesn't seem to improve on any of them besides using the name ponyxl for advertising purposes."
giving them credit is adverting?
should we have just stolen their prompting style without giving credit?
@DataVoid I appreciate your effort in pushing the technology forward, as I know that creating models is difficult and requires a lot of work. Thank you for your dedication.
I made a few specific observations in my initial comment:
1) Celebrities: Based on the images you've uploaded since my comment and the tone of your response, it seems that the difficulty with celebrities isn't an intentional safety feature. This suggests to me that adding details to prompts quickly degrades this model's ability to focus on likeness. While "Taylor Swift" produces an acceptable result, "Taylor Swift standing on stage at the Grammy's giving a speech" yields a random blonde woman with a disfigured face.
2) Prompt Coherence: If there's a specific prompting style that needs to be followed to help users understand what you're trying to demonstrate, more detail would be helpful. You mentioned in your comment and in previous versions of the post that this model follows the PonyXL style. As a frequent user of PonyXL, I find that my prompts do not produce coherent results with this model. Attempting to start from the short example prompts provided and add detail leads to unpredictable results.
3) Penis: Regarding your statement, "It can do artistic NSFW," I've observed that this model cannot generate a penis. Has anyone on your team successfully generated a penis on a human male using this model? If this limitation is not by design, it may indicate an insufficiently inclusive training and evaluation process. While many amateur models only train on waifus, I'm aware that you and your team are not amateurs based on your other models.
4) Using PonyXL for advertising: The updated listing addresses any concerns I had on this point. Thank you for being receptive to feedback.
5) Selling you something: The link out to https://rundiffusion.com/contact-sales led me to believe there might be a commercial aspect to this project. If this link is unrelated to your work, I'll edit my comment to remove that claim.
It can do celebrities. I wonder if you are aren't prompting it correctly.... as It worked for me first try. It is an experimental model.
Try this prompt with the other settings listed: Taylor Swift, source_women,score_9, score_8_up, score_7_up, score_6_up, score_5_up, score_4_up
@Colorblind_Adam in my follow up comment I expanded that it seems like you can't add details to the prompt without immediately degrading the likeness. your example prompt doesn't have any details in it.
Hmmm after more testing, im sorry for the initial rude comment. I seemed to have fixed my prompting styling to fit more in line with the way this model was trained.
I will still need to do more testing, it seems like this model is defying the traditional prompting methods; like pony...
Edit: Done testing, overall its a mediocre model. It feels like an SDXL model released in Oct 2023. Even thinkdiffusionXL released back in Oct 2023 has better prompt coherence. Still rough around the edges. I will be abandoning this model, as I am currently using a custom model merge I made myself that blows this out of the water.
My thumbs downs are not because its a bad model, its just that as of March 15 2024 there are so much better models out there.
What did you merge?
I think you run the risk of missing out on an exceptional model if you give up at this point. My initial view after re-using some of my old prompts was much the same as yours. I'm still experimenting though and getting some excellent results. Skin details for example are amazing. I think with a little more experience in learning how to 'drive' this model, you'll find it surpasses most of what is currently on the site.
(I'm not in any way affiliated with the author or Rundiffusion, just in case you're wondering)
@swedishViking Any tips to a noob who has no clue what prompt styles to use to attempt to discover something unique?
what models you think it's better than this model as you saying ?
Make sure you put a sample prompt or else people won't know to use CLIP SKIP 2 😅
excited to use this coming from pony and various mixes.
what sampler is this supposed to use?
DPM++ 2M Karras works well for me.
No joke, I've never seen a model give such a long winded introduction and then blow it on a few simple tips to ACTUALLY USE THE THING. Lol AND they failed to even include generation info in the cover pictures!
@SleezeBagDiffusion probably ai generated text
Where did 0.4 go? Can we rename 0.4 to 1.0 if we've downloaded 0.4?
You can still get V0.4 here: https://civitai.com/models/267242?modelVersionId=355900
I suspect they want to differentiate the two as this model is incorporating Rundiffusion
RTFM (if you want outstanding results)
CFG: 3 - 8.5 (Sweet Spot)
Steps: 30 - 40 (Higher creates artifacts)
Sampler: DPM++ 2M Karras
CLIP Skip: 2 (Very Important)
Positive Prompt: your prompt, score_9, score_8_up, score_7_up, score_6_up, score_5_up, score_4_up
Negative Prompt: Not Needed (or just add "deformed")
Please note that I'm not connected or affiliated with the developer of this model, or Rundiffusion, I'm simply supplying what they've put on the RD website
You won't typically be able to just plonk your old prompt and settings in and expect it to deliver the same result. This checkpoint requires a slightly different prompt syntax, clip skip and CFG settings. As always, experiment a little to see what works for you.
I've put some plots up to show different CFG values and their effects.
i thought we stopped using clip skip with sdxl (and all the sdxl derivatives)?
@yofoton174609 I'm not an expert on such matters, but my understanding is that this specific model has been developed under different conditions than 'normal' SDXL, and so the clip skip is once again relevant.
My initial testing is consistent with your thoughts though in that I'm not seeing any difference between different clip skip settings. That is likely because I've not yet mastered the prompting syntax for this model type so my 'normal SDXL prompts' are yielding normal SDXL clip-skip indifference.
Why pony style positive tags, if it's not a pony model? They specifically stated that in the end of the description.
Eww, pony tags. Please don't propagate those outside the regular ponydiffusion model family!
Is there a way to set Clip skip on just a per-image basis? or do I need to go in to A1111 settings, set it, restart the Web UI first?
@SleezeBagDiffusion If you have A1111/Web UI Forge with the Script extension, you can experiment with that.
😸👍
Just to clarify: this doesn't incorporate the virus that is ponydiffusion score_9 etc.? That style of prompting hearkens back to the days of SD 1.5 and booru tagging, and has (in my opinion) no place in an SDXL descendant, nor does it have a place in a (hopefully) future of natural-language prompting.
EDIT: Also, I see a bunch of comments saying to use CLIP skip 2, but the description says to use CLIP skip -2? Which is it?
In ComfyUI Clipskip -2 = 2 in Automatic 1111
I think keyword prompting is superior to natural language as you can be much more specific and take advantage of the right keywords.
+ not only curves but also straight, clear lines
+ drinks LORA dry
+ high accuracy of program execution
- requires experience and perseverance
= ideal high-end equipment
I read some of the comments and couldn’t pass by just leaving a like. Yes, I understand the model for God’s prompt, so to speak! BUT everything should be fair! Find me another model that is capable of not only drawing curved lines (photorealism, painting, daubing, etc., of which there are already hundreds), but also straight, clear lines (see my examples). Find me another model that drinks LORA dry. Find me a model with such high accuracy of program execution. In the meantime, I'm revoking my subscription to MJ and giving Buzz to the author. As for me, DataVoid decided to kill everyone at once.
thank you so much! this comment means the world to me!
with clip or without....?
This Model is.. 🍌's
This model is different from anything else out there. Innovative models are so much fun. It's capable of producing excellent images, and I'm having a blast figuring out how to do it consistently.
Will there be plans like Lightning, TURBO or LCM in the future?
Your model can be created with a higher CFG, (Lightning and other accelerated generation models due to low steps and low CFG lead to ineffective prompt words, support for artistic effects is very few)
is LCM the fastest of them three?