Back to see if this way of training works on a real person seems ok.

Start with Lora weight of around 0.5, move to more if her face is not showing enough.

Give it a try.