Hello and Welcome,

in this Step-By-Step guide I will introduce you to all needed informations, links, tools, Examples and down in the comments a Q&A session.

To start with me, I teach myself everything by myself. I will mention in this guide a few problems I had and how I solved them.

Abbreviations and Description:

Model: In machine learning, a model refers to a mathematical representation of a system trained to perform tasks like image generation or language processing. Different models for different styles.

• SDXL: Short for "Stable Diffusion XL." It is an advanced version of the Stable Diffusion model designed for high-quality image generation with larger and more complex datasets.

• Pony: Pony is a finetune of SDXL, all the other Pony checkpoint are variants/spins on that base finetune.

• Flux: A model focusing on character, style, or thematic generation. Flux is often used to experiment with creative prompts, character design, and storytelling through images.

• SD3.5: This represents a version of the Stable Diffusion model, possibly the third major release (v3.5). Each version typically brings improvements in image quality, speed, and model architecture.

Tags: (SDXL & Pony)
Tags are keywords or short phrases used to describe specific attributes, features, or elements within an image. They serve as metadata, helping the model understand and learn from the dataset during training. Tags are typically used in the context of image datasets to label various aspects like objects, styles, colors, or actions.

Example:

• Tags for an image: "sunset, beach, palm trees, vibrant colors, calm water"
  These tags describe the key elements in the image, guiding the model to recognize and associate them during training.

Prompts: (Flux & SD3.5)
Prompts are longer, descriptive phrases or sentences given to the model during inference (or use) to guide the generation of images. They are used to instruct the model on what to create or focus on when generating an image.

Example:

• Prompt for generating an image: "A serene sunset over a beach with palm trees swaying gently and vibrant colors in the sky."
  This prompt directs the model to generate an image based on the description provided.

LoRA (Low-Rank Adaptation):
LoRA is a fine-tuning method that adapts large models with fewer parameters, making it lightweight and efficient. It focuses on training a small subset of parameters, which reduces computational cost while allowing the model to learn specific styles, characters, or tasks. [Check CivitAI sektion Models-> LoRA]

Embeddings:
Embeddings are vector representations of words, concepts, or features in a continuous space. In image generation, embeddings map high-dimensional data (like text descriptions or image features) into a lower-dimensional space that the model can process efficiently. [Check CivitAI sektion Models-> Embeddings]

-> e.g.: positive prompt embedding:fFaceDetail = better image generation for face details; negative embedding:easynegative = to prevent e.g. bad face, low details, mutations, blurry images and so on.

Upscaler:
An upscaler is a tool used to increase the resolution of an image while enhancing its details. This is crucial in generating high-resolution outputs from models that initially produce lower-resolution images. Common upscaling techniques include algorithms like ESRGAN or Real-ESRGAN. [Check CivitAI sektion Models-> UpScaler]

-> e.g.: a low resolution image will be upscaled by its resolution = higher resolution with more details in the image.

VAE (Variational Autoencoder):
VAE is a type of neural network used for encoding data (like images) into a compact latent space and then decoding it back into an image. In the context of image generation models, VAEs help in generating high-quality images by providing a structured latent space representation. [Check CivitAI sektion Models-> VAE]

IP Adapter:
The IP (Image Processing) Adapter is a component that adjusts the image input or output to match certain requirements, such as resolution, color space, or feature alignment, before it is fed into or taken from the model. [Check Hugging Face]

-> e.g.: takes a skin colour from a image and but it on the new image = the new image will have the same skin colour.

ControlNet:
ControlNet is an advanced neural network that adds control layers to existing image generation models. It allows users to exert more control over specific aspects of image generation, such as style, structure, or content, by integrating additional guidance into the generation process. [Check Hugging Face]

-> e.g.: Takes a pose or body shape from a image and guide the new image with it = the new image will have the same pose or body shape.

Positive Prompt:
A positive prompt is a description or set of keywords that guide the model toward generating specific desired features in an image. It tells the model what elements, styles, or characteristics should be emphasized. For example, "a serene landscape with mountains and a clear blue sky" directs the model to create an image matching this description.

Negative Prompt:
A negative prompt, on the other hand, specifies what the model should avoid when generating an image. It helps to remove or reduce unwanted features or artifacts. For instance, "no people, no buildings, no bright colors" instructs the model to steer clear of these elements in the generated image.

Links:

OpenModelDB: A huge database for Upscaler with comparison

Hugging Face: A page to go for all kind of things, e.g.: models, vae, upscaler, missing scribts, cumfyUI nodes and way more.

Pinokio: 1-Click-installer for multiple purpose e.g.: Text2Speech, Text2Image or Image2Video.

ComfyUI: UI for perect control of any aspects of your creation.

Free ComfyUI Workflows: Openart offers free workflows and video tutorial to almoust all workflows.

Birme: Easy & free tool for resize / Crop images

FLUX Prompt Generator: Hugginface tool to creat randome Pompts or/and transform the prompt into a fitting flux prompt.

Flux.1 Prompt Optimizer: This Prompt Optimizer feature is on this webside free. [in case you wonna shorten the prompt.

Foreword:

In this guide I assume you know already what Image LoRA you wonna train and how you can creat images offline/online.

Thats why I will mention only a few software tools, what you can use for training your LoRA in Index 6. Train Lora.

Index:

1. Select Image

   E.g.: movie or cgi character, keep in mind that loras on real person should only used for SFW content.

   1. Face

   2. Upper body

   3. Full body

2. Select Model direction

   Select a model you wonna use later to creat images with your LoRA. Check the links to find it.

   1. Flux [Check CivitAI sektion Models-> Checkpoint]

   2. SDXL [Check CivitAI sektion Models-> Checkpoint]

   3. Pony [Check CivitAI sektion Models-> Checkpoint]

   4. SD3.5 [Check CivitAI sektion Models-> Checkpoint]

3. Create image with

   Generate more images to train your lora better.

   rule of thumb:

   • 20 images with

     • 10 Face, different angle

     • 5 upper body, different pose

     • 5 full body, different pose

   1. ControlNet

      1. Flux -> Attached file name:

         Flux image2ControlNet2image.json

         1. Pose

         2. Depth

         3. Canny

         4. Scribble

      2. Pony / SDXL -> Attached file name:
         SDXL ControlNet workflow.json
         Pony Controlnet.json

         1. Pose

         2. Depth

         3. Canny

         4. Scribble

      3. SD3.5 -> Attached file name: in V02

         1. Pose

         2. Depth

         3. Canny

         4. Scribble

4. (Optional) Upscale image

   1. 1x_DeBLR

   2. 4x-AnimeSharp

   3. 4x-ClearRealityV1

   4. 4xClearRealityV1_Soft

   5. 4xFaceUpDAT

   6. 4xUltrasharpV10

   7. 4_NMKD-Siax_200k

   8. 4_NMKD-Superscale-SP_178k

   9. 4_foolhardy_Remacri

   10. 8xNMKD_Faces_150k_G

   11. 8x_NMKD-Faces_160k_G

   12. DreamClear-1024 [New]

   13. ESRGAN_4x

   14. OmniSR_X2_DIV2K

   15. OmniSR_X3_DIV2K

   16. OmniSR_X4_DIV2K

   17. RealESRGAN_x2

   18. RealESRGAN_x4

   19. RealESRGAN_x4plus_anime_6B

   20. Nomos8khatlOtf_v20

   21. realesrGeneralx4_V3

   22. realesrganx4plus_V1

5. Resize image

   1. https://www.birme.net/ <- online

   2. GIMP Software <- offline

6. Create Prompts

   1. Offline:

      1. Workflow Joycaption -> Attached file name: Joy-Caption 2 Prompt.txt.json

      2. Workflow Florence2 -> Attached file name: Florence2 to Prompt.json

      3. Workflow WD-Tagger -> Attached file name: WD14-Tagger 2 Prompt.json

   2. Online:

      1. CivitAI feature: When you train your LoRA on CivitAI there can help you with ther Auto-Prompter for Tags (SDXL/Pony) or Captions (Flux/SD3.5)

      2. FLUX Prompt Generator

7. Train Lora

   1. CivitAI Online

   2. Pinokio Offline

      1. FluxGym

      2. Kohya_ss

8. Training Settings

   1. Epoch:
      An epoch refers to one complete pass through the entire training dataset. Training a model for more epochs allows it to learn more but can also lead to overfitting if done excessively.
      Example: Training with 5 epochs means the model will see the whole dataset 5 times.

   2. numRepeats:
      This parameter defines how many times each image or data point is repeated within a single epoch. Increasing numRepeats can help emphasize certain samples, especially when dealing with imbalanced datasets.
      Example: Setting numRepeats to 3 means each image is shown to the model three times per epoch.

   3. Resolution:
      The image resolution used for training. Higher resolutions can capture more detail but require more computational resources. Lower resolutions are faster but may lose finer details.
      Example: Training at 512x512 is common for many image generation models, but 1024x1024 may be used for high-quality outputs.

   4. ClipSkip:
      This parameter is used to skip a certain number of layers in the CLIP model during training, often to reduce memory usage or focus on different levels of abstraction.
      Example: Setting ClipSkip to 2 means the last two layers of CLIP are skipped during training.

   5. unetLR (UNet Learning Rate):
      The learning rate specifically for the UNet, which is the neural network responsible for generating the images. A smaller learning rate can lead to slower but more stable training.
      Example: A unetLR of 0.0001 is common for stable and effective training.

   6. lrScheduler (Learning Rate Scheduler):
      This adjusts the learning rate during training, either reducing it or increasing it based on the progress of the training process. It helps in fine-tuning the learning process and improving convergence.
      Example: Using a cosine annealing scheduler can gradually reduce the learning rate to zero as training progresses.

   7. optimizerType:
      The optimizer determines how the model updates its weights during training. Different optimizers can affect the speed and stability of learning.
      Example: AdamW is a popular optimizer for training image generation models due to its balance between performance and computational efficiency.

   8. Shuffle Caption:
      This setting determines whether captions (used as prompts for training) are shuffled or presented in a fixed order. Shuffling can prevent the model from overfitting to a specific order of data.
      Example: Enabling shuffle captions means the order of prompts changes in each epoch, ensuring the model generalizes better.

9. Training Settings Table !!Only my suggestion!!

   

10. LoRA Comparison

    1. Epoch Comparison

       1. SDXL/Pony

          1. Workflow attached: SDXL_Pony LoRA Epoch comparison.json

       2. Flux.D

          1. Workflow attached: Flux LoRA Epoch Comparison.json

    2. Steps vs. CFG Comparison

       1. SDXL/Pony

          1. Workflow attached: SDXL_Pony grid (cfg_steps).json

       2. Flux.D

          1. My other article for it

          2. Workflow attached: Flux+Lora grid (cfg_steps).json