WorkflowDataset creation:<ol><li>I've used gallery-dl (<a target="_blank" rel="ugc" href="https://github.com/mikf/gallery-dl">GitHub</a>) to scrape images of of different websites</li><li>I've removed duplicates with czkawka (<a target="_blank" rel="ugc" href="https://github.com/qarmin/czkawka">GitHub</a>)</li><li>I've chose my favorite images and captioned them using ChatGPT (gpt4-o)</li><li>I've captioned the rest with joycaption using this taggui fork (<a target="_blank" rel="ugc" href="https://github.com/doloreshaze337/taggui">GitHub</a>)</li><li>I've used [Code 1] to make sure the captions were made correctly using an llm with ollama</li><li>I've trained the model for 6000 steps using [config 1] with ostris/ai-toolkit (<a target="_blank" rel="ugc" href="https://github.com/ostris/ai-toolkit">GitHub</a>)</li></ol>Code 1:<pre><code>import ollama
import os

directory = [dataset-path]
instruction = """You will receive a caption text that describes pictures featuring a the game character from Bloodborne called the "Hunter". Your task is to seamlessly incorporate the characters name into the caption with minimal changes when the characters name is not mentioned. The goal is to subtly include the characters name without explicitly stating that the character is called the "Hunter" if this is not the case already. If the characters name is already mentioned in the caption just return the caption as it was. Do not return something else besides the caption"""
modelname = "Tiger-Gemma-9B:latest"
output_directory = [output-path]

def pairCaptions(txtfiles, first_directory, instruction, modelname, output_directory, second_directory):
 progress = 0
 for txtfile in txtfiles:
 with open(os.path.join(first_directory, txtfile), "r") as content: caption = content.read()
 with open(os.path.join(second_directory, txtfile), "r") as content2: caption2 = content2.read()
 modifiedcaptions = ollama.generate(model=modelname, prompt=f"{instruction}{caption}. And here is the descriptory list: {caption2}")
 open(os.path.join(output_directory, txtfile), "w").write(modifiedcaptions["response"])
 progress += 1
 print(f"Modified {txtfile} ({progress}/{len(txtfiles)})")
 print(modifiedcaptions["response"])

def CaptionsEdit(txtfiles, directory, instruction, modelname, output_directory):
 progress = 0
 for txtfile in txtfiles:
 with open(os.path.join(directory, txtfile), "r") as content: caption = content.read()
 modifiedcaptions = ollama.generate(model=modelname, prompt=f"{instruction}This is the caption text: {caption}")
 open(os.path.join(output_directory, txtfile), "w").write(modifiedcaptions["response"])
 progress += 1
 print(f"Modified {txtfile} ({progress}/{len(txtfiles)})")
 print(modifiedcaptions["response"])

txtfiles = [f for f in os.listdir(directory) if f.endswith(".txt")]
CaptionsEdit(txtfiles, directory, instruction, modelname, output_directory)</code></pre>Config 1:<pre><code>job: extension
config:
 name: f1_char_hunter_v2
 process:
 - type: sd_trainer
 training_folder: output
 device: cuda:0
 network:
 type: lora
 linear: 16
 linear_alpha: 16
 save:
 dtype: float16
 save_every: 319
 max_step_saves_to_keep: 4
 push_to_hub: false
 datasets:
 - folder_path: [path]
 (bloodborne)\hunterbb_v2
 caption_ext: txt
 caption_dropout_rate: 0.05
 shuffle_tokens: false
 cache_latents_to_disk: true
 resolution:
 - 512
 - 768
 - 1024
 train:
 batch_size: 1
 steps: 6000
 gradient_accumulation_steps: 1
 train_unet: true
 train_text_encoder: false
 gradient_checkpointing: true
 noise_scheduler: flowmatch
 optimizer: adamw8bit
 lr: 0.0001
 ema_config:
 use_ema: true
 ema_decay: 0.99
 dtype: bf16
 model:
 name_or_path: black-forest-labs/FLUX.1-dev
 is_flux: true
 quantize: true
 sample:
 sampler: flowmatch
 sample_every: 319
 width: 1024
 height: 1024
 prompts:
 - The image is a digital drawing rendered in a dark moody style with a muted
 color palette dominated by shades of brown and grey. The character stands
 in a commanding pose clad in a long dark coat with intricate ornate details
 reminiscent of a historical military uniform. The coat is adorned with silver
 buttons and a high collar and a large dark hat with a feathered plume sits
 atop the character's head casting a shadow over their face. The hunter's facehunter
 - Hunter stands in a powerful, poised stance, dressed in dark, worn armor with
 detailed engravings on the bracers and boots. The character's face is obscured
 by a high-collared scarf and a battle-worn helmet with broken edges. Hunter
 wields a large, serrated blade in one hand and a finely crafted pistol in
 the other. The muted, smoky background enhances the grim and determined aura
 of the figure.
 - Hunter is depicted in a dynamic pixel art style, standing in an aggressive,
 battle-ready stance. The character's flowing cape and dark attire are accentuated
 with bold shading and highlights, giving depth to the design. Hunter holds
 a large, serrated weapon in one hand and a gun in the other, both drawn in
 intricate pixel detail. The overall composition, framed by a thin border,
 captures a sense of action and intensity.
 - Hunter stands in a mystical and foreboding scene, surrounded by colossal,
 shadowy statues of beasts emerging from swirling orange and blue mists. The
 character is dwarfed by these monumental figures, yet their posture remains
 calm and composed, holding a weapon at their side. A large, glowing moon looms
 behind a domed structure, casting an ethereal light over the surreal landscape,
 creating a vivid contrast between light and shadow, heightening the atmosphere
 of mystery and tension.
 - Hunter is captured in a moment of intense agony or transformation, with their
 head thrown back and hands clawing upward. The character's form is engulfed
 in darkness, contrasting sharply with the vivid, fiery red background that
 swirls around them. Their eyes glow with a fierce, almost demonic light, and
 the scene exudes a powerful sense of torment and rage. The dramatic use of
 color and the dynamic pose create a haunting, visceral image of Hunter in
 the throes of a violent and overwhelming force.
 - The image is a digital drawing in a dark gothic style featuring a male character
 with a stern imposing presence. The character is dressed in a long dark coat
 with intricate stitching and a high collar adorned with a large dark hat that
 covers most of his face giving him a mysterious intimidating appearance. His
 eyes are hidden behind the hat adding to the enigmatic aura. The coat is detailed
 with gold accents on the shoulders and cuffs and a large belt buckle.hunter
 - The image is a digital drawing in an anime style depicting a young woman with
 a pale complexion and long lavender hair styled in a braid with a blue ribbon.
 She wears a black wide-brimmed hat with a dark almost black coat and gloves
 giving her a mysterious and ominous appearance. Her eyes are large and expressive
 with a hint of sadness or determination. The background is a fiery scene with
 dark swirling smoke and flames suggesting a battle or a post-aphunter
 - Hunter is illustrated in a stylized, semi-realistic portrait, with a focused
 side profile. The character has striking white hair partially concealed by
 a dark, tattered hat, while their lower face is obscured by a black scarf.
 Hunter's piercing blue eyes stand out against the muted tones of their green
 cloak and dark attire. The clean, minimal background enhances the character's
 sharp, detailed features, conveying a sense of calm determination and mystery.
 neg: ''
 seed: 99
 walk_seed: true
 guidance_scale: 4
 sample_steps: 20
meta:
 name: f1_char_hunter_v2
 version: '2.0'</code></pre>PS: If you want the dataset please contact me. I just don't want to get CivitAI in copyright trouble.

Files with SHA256: 9154c6aeb483f584e45db03f609fa163df12632079404bcba721f26d277a8bef