Workflow

Dataset creation:

1. I've used gallery-dl (GitHub) to scrape images of of different websites

2. I've removed duplicates with czkawka (GitHub)

3. I've chose my favorite images and captioned them using ChatGPT (gpt4-o)

4. I've captioned the rest with joycaption using this taggui fork (GitHub)

5. I've used [Code 1] to make sure the captions were made correctly using an llm with ollama

6. I've trained the model for 6000 steps using [config 1] with ostris/ai-toolkit (GitHub)

Code 1:

import ollama
import os

directory = [dataset-path]
instruction = """You will receive a caption text that describes pictures featuring a the game character from Bloodborne called the "Hunter". Your task is to seamlessly incorporate the characters name into the caption with minimal changes when the characters name is not mentioned. The goal is to subtly include the characters name without explicitly stating that the character is called the "Hunter" if this is not the case already. If the characters name is already mentioned in the caption just return the caption as it was. Do not return something else besides the caption"""
modelname = "Tiger-Gemma-9B:latest"
output_directory = [output-path]

def pairCaptions(txtfiles, first_directory, instruction, modelname, output_directory, second_directory):
    progress = 0
    for txtfile in txtfiles:
        with open(os.path.join(first_directory, txtfile), "r") as content: caption = content.read()
        with open(os.path.join(second_directory, txtfile), "r") as content2: caption2 = content2.read()
        modifiedcaptions = ollama.generate(model=modelname, prompt=f"{instruction}{caption}. And here is the descriptory list: {caption2}")
        open(os.path.join(output_directory, txtfile), "w").write(modifiedcaptions["response"])
        progress += 1
        print(f"Modified {txtfile} ({progress}/{len(txtfiles)})")
        print(modifiedcaptions["response"])

def CaptionsEdit(txtfiles, directory, instruction, modelname, output_directory):
    progress = 0
    for txtfile in txtfiles:
        with open(os.path.join(directory, txtfile), "r") as content: caption = content.read()
        modifiedcaptions = ollama.generate(model=modelname, prompt=f"{instruction}This is the caption text: {caption}")
        open(os.path.join(output_directory, txtfile), "w").write(modifiedcaptions["response"])
        progress += 1
        print(f"Modified {txtfile} ({progress}/{len(txtfiles)})")
        print(modifiedcaptions["response"])

txtfiles = [f for f in os.listdir(directory) if f.endswith(".txt")]
CaptionsEdit(txtfiles, directory, instruction, modelname, output_directory)

Config 1:

job: extension
config:
  name: f1_char_hunter_v2
  process:
  - type: sd_trainer
    training_folder: output
    device: cuda:0
    network:
      type: lora
      linear: 16
      linear_alpha: 16
    save:
      dtype: float16
      save_every: 319
      max_step_saves_to_keep: 4
      push_to_hub: false
    datasets:
    - folder_path: [path]
        (bloodborne)\hunterbb_v2
      caption_ext: txt
      caption_dropout_rate: 0.05
      shuffle_tokens: false
      cache_latents_to_disk: true
      resolution:
      - 512
      - 768
      - 1024
    train:
      batch_size: 1
      steps: 6000
      gradient_accumulation_steps: 1
      train_unet: true
      train_text_encoder: false
      gradient_checkpointing: true
      noise_scheduler: flowmatch
      optimizer: adamw8bit
      lr: 0.0001
      ema_config:
        use_ema: true
        ema_decay: 0.99
      dtype: bf16
    model:
      name_or_path: black-forest-labs/FLUX.1-dev
      is_flux: true
      quantize: true
    sample:
      sampler: flowmatch
      sample_every: 319
      width: 1024
      height: 1024
      prompts:
      - The image is a digital drawing rendered in a dark moody style with a muted
        color palette dominated by shades of brown and grey. The character stands
        in a commanding pose clad in a long dark coat with intricate ornate details
        reminiscent of a historical military uniform. The coat is adorned with silver
        buttons and a high collar and a large dark hat with a feathered plume sits
        atop the character's head casting a shadow over their face. The hunter's facehunter
      - Hunter stands in a powerful, poised stance, dressed in dark, worn armor with
        detailed engravings on the bracers and boots. The character's face is obscured
        by a high-collared scarf and a battle-worn helmet with broken edges. Hunter
        wields a large, serrated blade in one hand and a finely crafted pistol in
        the other. The muted, smoky background enhances the grim and determined aura
        of the figure.
      - Hunter is depicted in a dynamic pixel art style, standing in an aggressive,
        battle-ready stance. The character's flowing cape and dark attire are accentuated
        with bold shading and highlights, giving depth to the design. Hunter holds
        a large, serrated weapon in one hand and a gun in the other, both drawn in
        intricate pixel detail. The overall composition, framed by a thin border,
        captures a sense of action and intensity.
      - Hunter stands in a mystical and foreboding scene, surrounded by colossal,
        shadowy statues of beasts emerging from swirling orange and blue mists. The
        character is dwarfed by these monumental figures, yet their posture remains
        calm and composed, holding a weapon at their side. A large, glowing moon looms
        behind a domed structure, casting an ethereal light over the surreal landscape,
        creating a vivid contrast between light and shadow, heightening the atmosphere
        of mystery and tension.
      - Hunter is captured in a moment of intense agony or transformation, with their
        head thrown back and hands clawing upward. The character's form is engulfed
        in darkness, contrasting sharply with the vivid, fiery red background that
        swirls around them. Their eyes glow with a fierce, almost demonic light, and
        the scene exudes a powerful sense of torment and rage. The dramatic use of
        color and the dynamic pose create a haunting, visceral image of Hunter in
        the throes of a violent and overwhelming force.
      - The image is a digital drawing in a dark gothic style featuring a male character
        with a stern imposing presence. The character is dressed in a long dark coat
        with intricate stitching and a high collar adorned with a large dark hat that
        covers most of his face giving him a mysterious intimidating appearance. His
        eyes are hidden behind the hat adding to the enigmatic aura. The coat is detailed
        with gold accents on the shoulders and cuffs and a large belt buckle.hunter
      - The image is a digital drawing in an anime style depicting a young woman with
        a pale complexion and long lavender hair styled in a braid with a blue ribbon.
        She wears a black wide-brimmed hat with a dark almost black coat and gloves
        giving her a mysterious and ominous appearance. Her eyes are large and expressive
        with a hint of sadness or determination. The background is a fiery scene with
        dark swirling smoke and flames suggesting a battle or a post-aphunter
      - Hunter is illustrated in a stylized, semi-realistic portrait, with a focused
        side profile. The character has striking white hair partially concealed by
        a dark, tattered hat, while their lower face is obscured by a black scarf.
        Hunter's piercing blue eyes stand out against the muted tones of their green
        cloak and dark attire. The clean, minimal background enhances the character's
        sharp, detailed features, conveying a sense of calm determination and mystery.
      neg: ''
      seed: 99
      walk_seed: true
      guidance_scale: 4
      sample_steps: 20
meta:
  name: f1_char_hunter_v2
  version: '2.0'

PS: If you want the dataset please contact me. I just don't want to get CivitAI in copyright trouble.