<h1 id="where-mix-ups-come-from"> Where mix-ups come from </h1> Diffusion models connect words to image areas through **cross-attention**. At every step, every position of the image "looks at" the prompt's words and takes in some of each. If "dog" and "cat" pull on the same positions, the model draws a mixture: a cat-dog, a dog's head on a person, or colours that land on the wrong thing ("purple" meant for the teapot ends up on the apron). You can see this attention directly. The<a rel="ugc" href="https://github.com/WifiLast/sd-webui-act-steer"> act-steer</a> extension for Forge records, for every word, how much of the attention each image area gave it, and shows it as a **word map** above the image: - **all words**: every area is tinted in the colour of the word that dominates it. The coloured chips under the image are the prompt's words in their colours; grey words are filler words. - **one word** (hover or click a chip): a spotlight. The word's area lights up in its colour and the rest of the image is darkened. *Spotlight on the word "dog". Without the LoRA its attention sits on a head attached to a person; with the LoRA it covers a dog of its own.* From these maps the extension computes how much two words **overlap** (do they claim the same area?). Its *Optimize* button already uses that to adjust word strengths for one prompt at a time. The idea of the LoRA: **learn that fix once, inside the model, so it applies to every prompt.** <h1 id="the-idea"> The idea </h1> The LoRA changes only the two parts of cross-attention that decide *where* a word goes: - the **query** projection (what an image position is looking for), and - the **key** projection (how a word presents itself). It does **not** touch the **value** and **output** projections, which decide *what* a word paints. So "red" still means the same red and "watercolor" the same style; only the layout of which word acts where can change. Training never sees pixels either: it only looks at attention maps, so it can't pick up a new style from its training images.

Files with SHA256: 1b60dde09b88999612efb903b13ee5c90bc193e5a3ea987198d26872b5b2e1a1