While the built in embedding for Emma Stone is not terrible, I was curious on whether I could improve that.
I used 443 sample images, all cropped and tagged manually, mostly chosen from the top 1000 of the posts in her subreddit.
Description
1 vector. I may try training more vectors to add onto this to see if it improves any further, but I was surprised to see how good this looked with one vector alone. I think it already looks better than the what you get when you type Emma Stone into the base model, and the base model uses 2 vectors!
Trained on a 3080Ti (12GB)
Batch Size 24, 20,000 steps. Custom learning rate schedule:
8e-3:40, 7e-3:160, 6e-3:320, 5e-3:520, 4e-3:780, 3e-3:1140, 2e-3:1720, 1e-3:2300, 9e-4:2440, 8e-4:2620, 7e-4:2820, 6e-4:3060, 5e-4:3360, 4e-4:3760, 3e-4:4320, 2e-4:5320, 1e-4:6340, 9e-5:6620, 8e-5:6960, 7e-5:7340, 6e-5:7820, 5e-5:8440, 4e-5:9260, 3e-5:10420, 2e-5:12160, 1e-5:13560, 9e-6:13860, 8e-6:14200, 7e-6:14560, 6e-6:14960, 5e-6:15400, 4e-6:15920, 3e-6:16560, 2e-6:17420, 1e-6:18100, 9e-7:18260, 8e-7:18440, 7e-7:18620, 6e-7:18840, 5e-7:19080, 4e-7:19380, 3e-7:19760, 2e-7
FAQ
Comments (6)
This is downright frightening.
I don't understand
@fudefrak Your embedding is scary looking. It resembles Emma Stone, but it's extremely creepy.
That being said, I'd like to know why you used 400+ source images when every single guide, tutorial, and video about Textual Inversions recommend about 20. Also, where did you get your biblically long training schedule? Again, no guide nor overview of the process recommends something anywhere near that complex. Finally, did you check your vector strength during generation?
@Bozack3000 I'm not sure what you find creepy about any of these images. Compared to real images, what specifically do you see about them that looks like it stands out to you? I mean obviously there are some areas that the base SD 1.5 does poorly at with the default 512x512 resolution, but anything other than that? As I said in the description, I am willing to try training additional vectors to see what they might change or improve, but I was pretty surprised at just how good it was able to get with 1 vector alone. Seems to work well with other models too from what I can tell.
Guides focus on trying to come up with something usable in as little time as possible. I wanted to focus on coming up with something as accurate as I can. The more images you have, the more variety the training process has to pick up on, the less it focuses too strongly on any particular characteristic of specific images, but of the overall concept. Pretty much any deep learning will work best with more data to learn from.
My schedule is something I came up with over a lot of experimentation. It allows for a very gradual change in learning rate over the entire process, which I've found is better than sudden changes at various steps. If I could use a formula instead, I would.
The curve I came up with was based on a more simplified schedule I was using earlier:
5e-3:1000, 5e-4:5000, 5e-5:12000, 5e-6:18000, 5e-7:20000
Which focuses on giving most of its time to the middle sections, with less time spent on the highest and lowest values
Using the midpoints between each segment (500, 3000, 8500, 15000, 19000) and the exponents for each, I came up with a cubic trendline that somewhat went through each of those points, applied 5*10^value, and then broke it down into 1 significant figure, at 20 step intervals, and that gave me my final learning schedule. It ended up working a LOT better than the above simplified version. I also scale this schedule to different step totals, based on batch size, which will vary between SD 1.5 and 2.1, and I also use smaller numbers for hypernetworks but otherwise follow the same pattern.
@fudefrak Thanks for the explanation, I guess the results speak for themselves 🙂