I was not sure what happened with v.1, but I struggled to make consistent images with it, so I just remade the whole thing only to discover the same problems in v. 2.
With v. 3, it works much better, but for some reason, this is still a face doesn't look quite right sometimes.
Check the weight of the lora and find the one that best works with whatever checkpoint you're using.