Trying To Train a Complicated LoRA, I Am Running Into Problems Please Help

Cross Posted from Reddit

As the title states, I am trying to train a lora for a complicated concept. It is from a manga called shadows house where there are shadow people and living dolls who mimic their emotions. These dolls also wear special uniforms, which I want to include in the lora. Also importantly the shadows wear different clothes and not the same uniform as the dolls.

Example

And here is my captions for it

Shadow and faceF, high collar, frilled collar on doll, blue dress on doll, black gloves on doll, black stockings on doll, red bow on doll, cropped, candle, stone wall, brown hair, green eyes, smile, holding own hands, curly hair, long hair, orange dress on shadow, flowing sleeves, short dress, bush, stone path, grass

My biggest problem is inconstancy, where pieces of the doll uniform will show up on the shadow or vice versa. I want the uniform to only show up on the doll, so how do I solve this problem?