I think the lora is too strong, use it with less strength like 0.67 ( ͡~ ͜ʖ ͡°), even 0.5