My Journey into Flux LoRA training.

Over the last few weeks I've been reading up on how to train a FLUX LoRA. There are 1001 articles all saying similar things but no real definitive easy to follow guide for this. There is also 101 different theories about what makes the better LoRA. This article and it's images demonstrates one of those theories... The "Train with Dev vs train with a finetuned model.

Models used:-

For FluxDEV I'm using the standard model from Black Forest Labs - https://huggingface.co/black-forest-labs/FLUX.1-dev

For the FluxDev Finetuned model I'm using AbdallahAlswa80's model - https://civitai.com/models/812892/flux1-dev2pro

The reason is this was the first article I came across on FluxDEV training. This is all in-house training on my workstation using the ComfyUI FluxTraining nodes - https://github.com/kijai/ComfyUI-FluxTrainer - I did customize the workflow a little to increase the number of epochs in a single training session.

Training Settings :-

These are pretty standard and defaulted from the recommendation shown on the node docs. Apart from Steps and "Epochs". There are 80 images from a video game that I very much like the art style from (Disclaimer note, It is free to download and play from the developers, and it is a NSFW game) I chose this to run tests because the art style is somewhat unique and is perfect for testing a wide array of situations.

The image dataset is 80 images mostly of completely different images but some similar images with different things going on in the room or outside etc.

The tagging is a single word (I plan on testing Single word with phrasing in another test) this test is purely to determine if training for a finetuned model vs Dev Base (distilled) model.

Steps per Epoch is 125. I had run many test between low count steps and high count steps and determined 125 seems to produce subtle changes between the saves.

Epochs - 20. This makes the total steps for a single image 2500 over 20 epochs. But over the entire dataset it is 200,000 Steps over 20 epochs. I did have a 480,000 Steps LoRA (80 images, 75 steps per, 80 epochs) and when I wanted to run this test I cut it in half thinking it'll give me a good idea.

Findings :-

The images can speak for themselves. I've Zipped up each set, along with the base LoRA Less DEV checkpoint from Black Forest Labs render for the same prompt. Along with a reference image of the art style.

Since I am limited to 20 images and I have 82 of them, I'm taking a handed picked distribution of images across the entire 2500 steps. The choosing was "any significant changes" there were several steps that produces somewhat similar images to each other. Those were skipped.

Each image will have the steps and the Checkpoint used in the prompt. (No where else to put this)

I feel that while a finetuned model may have it's application, when it comes to replicating art style it suffered from "drawing thicker lines" While that is in itself an art style, it doesn't appear re-fined enough. That said the more and more Dev trained the more abominations started leaking through long before it started happening on the finetuned model. Notice the last images in the set, the Dev model really had issues with the chair. While the finetuned also started 'disfiguring' the chair, you could dismiss that as "artist style" rather than errors.

Another note here - Sometimes the finetuned produces some major changes quicker than the Dev. A good example of this is the Bride facing the other way. The Finetuned started facing the bride the other way, 1 epoch earlier than Dev.

My favorite from both checkpoints:-

Dev - 1750 Steps

Finetuned - 1375 Steps