UPDATE: Following the common practice, the most recent version is trained on NAI with clip skip 2
The advantage of my models is clearly the ability to put multiple characters in a same image
(I was considering training a full model with the same dataset as well, but the LoHa seems to be reasonably good at this point
The other models are otherwise trained on ACertainty
Please check the version notes for further details including the clip skip to use
Normally you should be able to use the lycoris models on a lot of different checkpoints. In case of incompatibility, you can lower lycoris weight or better, merge your base model with AC/nai.
More checkpoints can be found on the associated hugging face repository https://huggingface.co/alea31415/oshinoko
Screenshots and fan arts are respectively tagged with aniscreen and fanart. Using aniscreen gives both stronger anime style and better resemblance. Using aniscreen on style models may cause weird effects.
You may want to use ; to separate characters (see the prompts of the example images)
Characters
EP01
HoshinoAi (add hair ornament if you want)
AmemiyaGoro
TendojiSarina
Aquamarine baby
Aquamarine child
Aquamarine
Ruby baby
Ruby child
Ruby (add red ribbon if you want)
ArimaKana child
ArimaKana (I suppose you should add some hat)
SaitoIchigo
Miyako
GotandaTaishi
KurokawaAkane
Main improvement of EP02
Aquamarine
Ruby
Added in EP04
NarushimaMelt
KaburagiMasaya
KichijoujiYoriko
ShiranuiFrill
KotobukiMinami
Added at the end of the season
MemCho
KumanoNobuyuki
SumiYuki
MorimotoKengo
Pieyon
Comments on the most recent model:
Pieyon is not learned very well as can be seen from the example images, showing this is a more difficult task than learning the appearance of the characters
The star-shaped eyes are learned pretty well in the newest model, probably too well that it sometimes wants to put it on other characters, especially when we do inpainting
You may want to avoid this by putting related keywords in negative
I suppose this suggests a sort of character blending
Someone reported to me the EP04 version has more important character blending compared to the EP02 version
I am not sure how it would be the case for this new one as I do not have an objective way to judge this
Hopefully it works fine for you
Dataset composition
fanart 2432
screenshots
EP 01 1268
EP 02 357
EP 03,04 777
EP 05-11 2582
regularization ~35K
--
I don't need support or credit, but I would be glad to know that you are using the models I trained and find it useful.
Moreover, I would like to advocate for more franchise models.
You can take a look at my workflow https://github.com/cyber-meow/anime_screenshot_pipeline if you are interested.
You may also check this nice page https://civarchive.com/articles/262/directory-of-girlpacks-character-packs for a collection of such models,
Description
Trained with clip skip 2
Characters with fewer images can be more sensitive to the choice of base models
You can consider using checkpoints of other training steps as well https://huggingface.co/alea31415/oshinoko/tree/main/checkpoints_0505ep04 (see the last two images for some comparison)
FAQ
Comments (19)
I collected the images of each character in a file to make it easier for you to use. I saved them without a background
See the file. Can you create lora ?
https://mega.nz/folder/YK8VAYwK#DmnYUIMBgHNcb0q7xkYgXQ
I found that you are the only person who is good at creating Ensemble Stars characters
Thank you
Provided that you already have the dataset I can of course do i since all I need to do is to launch the scripts.
However I do not know Ensemble Stars personally so I will not be able to judge the quality of the results and I won't generate example images myself.
It will also be better if you can put all of them in a zip file because otherwise the download is slow for me.
Finally, it is worth noticing that as you only have single-character images you will have difficulty generating multi-character images.
@alea31415 It's okay, the important thing is that lora make me these characters if possible
Do you want any other information? I put each character in a file so that you can know the character
@tamaokaima Here you go https://huggingface.co/alea31415/ensemble-stars-characters/tree/main/loha_05082023
I trained some LoHas at clip skip 2
Having all the backgrounds removed seems to cause some weird effect https://i.imgur.com/QdwcNDu.jpg
However I think it can be more or less fixed if you prompt the background properly
https://i.imgur.com/TUiJyVc.png https://i.imgur.com/fAoXzpS.png
As I said earlier I did not test it extensively
Feel free to share it with others or otherwise I can also put it on CivitAI if you are willing to provide some example images
@alea31415 Thank you for everything you have done. It seems that there is a problem with the background that makes the pictures strange and distorted
But with that said, I thank you for your help and for your work. I have a question instead, could you work for lora for this
https://mega.nz/folder/FHMxlIDI#iC25B8KMsCiJTU5JTHvStQ
It seems that you will succeed in creating these characters because they look much easier
See the file
@tamaokaima It seems you are trying to do a style Lora here.
Every image has different characters so it doesn't seem to be suitable for learning characters.
Since there are only 58 images the training should be fast and you can probably try to figure out how to do it on colab yourself. In that case you can also retrain in case you are not satisfied.
Otherwise I may try to take care of it tonight if I have time (European time zone).
What a great work you have done! Giving thanks to you for bringing all these my favorite characters! One question by the way, the full ckpt is about 2.0G, what is it exactly? Like part of the unet or something else? And the latter locon or loha stuff is just extracted from it or further finetuned with each eposide?
Thanks for your kind words.
Full ckpt is the whole model in fp16 format. It is the version trained on the first episode. Locon is extracted from it.
Lohas are trained independently.
I'm quite busy now so I plan only to update near the end of season. I may again train the two for that.
@alea31415 Ok, f16 will make sense and thanks again for your great work!!!
just want to let you know that ep2 is much more accurate with characters. episode4 one bleeds way too much from other characters even when you put everyone else in the negative
Interesting. I didn't notice it when I tested myself. Can you give me some evidence or clarify what you mean by blending (do you think it is the case for my example images)? Maybe this happens for some specific base model?
I guess it will be fine if I train the entire model. Intuitively I would also think increasing Loha dimension can solve the problem, but maybe this is more subtle. Alternatively you can try the version with more steps to see if this is just because the model is undertrained.
@alea31415 hair styles, eye color,hair ornament and other stuff bleeds into the generation from other characters.Tried it on 3 models with 4 char and ep2 version was better in character fidelity in all of them.
@Wizi Thanks for the information. As far as I remember the captions are generated in the same way, so I guess this may be an issue of undertraining.
@alea31415 ^_^ just wanted to help as much as i can. Appreciate the good work o7
Hello, when I use this LYCORIS, Aquamarine and Ruby always have the same two eyes, and I cannot make their single eye have a star symbol. How can I solve this problem
This is subtle. In my case I generally do no have problems with Ruby, but it is more difficult for Aqua. Indeed, having the star on the correct eye is the hardest thing to learn on this model, so I think this property may be quiet easily lost when you switch base model.
Note that if you put star in the prompt it seems that you always get two stars (so don't put it).
The obvious solution is of course to do inpainting (masked only) after removing star from one eye manually.
Otherwise you just think of them as Ruby and Aqua from the most recent chapter from mangas :)
Sorry I don't really have a perfect solution here.
@alea31415 It's okay, after removing the star prompt, I can get the correct eyes. Thank you for your suggestion!
@alea31415 Excuse me, when I'm drawing Tendoji Sarina, she always has hair, how can I avoid it
@Zirend That's weird. Maybe you can try to put hair in negative?
Anyway I will train a new version soon as the season comes to an end.
Hopefully it can be better in some aspects.
Details
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.