SDXL lora trained on SD1.5. Trained on 'headshot of person' where person is a wildcard. update much smaller lora + cleaner dataset (2671 filtered images) + trained longer (8025 steps at batch size 16) Larger images at <a target="_blank" rel="ugc" href="https://ntcai.xyz/sdxl/headshot">https://ntcai.xyz/sdxl/headshot</a>Usage:<a target="_blank" rel="ugc" href="https://ntcai.xyz/comfyui/lorasimplesdxl.json">https://ntcai.xyz/comfyui/lorasimplesdxl.json</a> comfyuiQuick note: Support us and receive an exclusive NFT airdrop as a token of our appreciation. <a target="_blank" rel="ugc" href="http://patreon.com/NTCAI">patreon.com/NTCAI</a> This process is:<ol><li>Use img2img on SDXL output using SD1.5. I wrote a tutorial on this here: <a target="_blank" rel="ugc" href="https://civarchive.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui">https://civarchive.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui</a> ( <a target="_blank" rel="ugc" href="https://ntcai.xyz/articles/applying_stable_diffusion_15_models_loras_controlnet_to_sdxl_with_comfyui_and_img2img/">mirror</a> ) Create a large dataset with this technique. I ran this overnight on two A6000s. Hint: choose a subject matter that sd1.5 knows well and perhaps reject any distorted images.</li><li>Get your files in the correct form. This tutorial helped me:<div data-youtube-video><iframe allowfullscreen="true" autoplay="false" disablekbcontrols="false" enableiframeapi="false" endtime="0" ivloadpolicy="0" loop="false" modestbranding="false" origin playlist src="https://www.youtube.com/embed/AY6DMBCIZ3A" start="0" width="640" height="480"></iframe></div></li><li>Train SDXL using <a target="_blank" rel="ugc" href="https://github.com/kohya-ss/sd-scripts">https://github.com/kohya-ss/sd-scripts</a> on the generated images.<pre><code># Full command
CUDA_VISIBLE_DEVICES=0 accelerate launch --num_cpu_threads_per_process=2 "sdxl_train_network.py" --enable_bucket --pretrained_model_name_or_path="/ml2/trained/ComfyUI/models/checkpoints/stable-diffusion-xl-base-1.0/sd_xl_base_1.0.safetensors" --train_data_dir="/ml2/trained/sd-scripts/data/headshot" --resolution="1024,1024" --output_dir="/ml2/trained/ComfyUI/models/loras/Lora/sdxl" --logging_dir="./logs" --save_model_as=safetensors --output_name="headshot2" --network_alpha="1" --network_dim=32 --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --lr_scheduler="constant" --train_batch_size="16" --max_train_steps="10000" --mixed_precision="bf16" --save_every_n_epochs="1" --save_precision="bf16" --caption_extension=".txt" --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale</code></pre></li></ol> My hope is that this can help creators migrate their work. Happy training! Resources used:<ul><li>eyes lora <a target="_blank" rel="ugc" href="https://civarchive.com/models/72447/eyes-trained-with-no-data-trick">https://civarchive.com/models/72447/eyes-trained-with-no-data-trick</a></li><li>SD1.5 aniverse <a target="_blank" rel="ugc" href="https://civarchive.com/models/107842">https://civarchive.com/models/107842</a></li><li>person wildcards <a target="_blank" rel="ugc" href="https://civarchive.com/models/98622/clonecleaner-wildcards-increase-character-consistency">https://civarchive.com/models/98622/clonecleaner-wildcards-increase-character-consistency</a></li></ul>

Headshot - SDXL lora trained on SD1.5 output

Trained for 8025 steps at batch size 16 on the phrase ' headshot of person ' where person is a wildcard. The phrase was run through the SDXL to SD1.5 pipeline outlined here <a target="_blank" rel="ugc" href="https://civitai.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui">https://civitai.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui</a><pre><code># Full command
CUDA_VISIBLE_DEVICES=0 accelerate launch --num_cpu_threads_per_process=2 "sdxl_train_network.py" --enable_bucket --pretrained_model_name_or_path="/ml2/trained/ComfyUI/models/checkpoints/stable-diffusion-xl-base-1.0/sd_xl_base_1.0.safetensors" --train_data_dir="/ml2/trained/sd-scripts/data/headshot" --resolution="1024,1024" --output_dir="/ml2/trained/ComfyUI/models/loras/Lora/sdxl" --logging_dir="./logs" --save_model_as=safetensors --output_name="headshot2" --network_alpha="1" --network_dim=32 --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --lr_scheduler="constant" --train_batch_size="16" --max_train_steps="10000" --mixed_precision="bf16" --save_every_n_epochs="1" --save_precision="bf16" --caption_extension=".txt" --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale</code></pre>Reducing the network_dim reduced the filesize

headshot2.safetensors

v2.0

v1.0

&lt;p&gt;This is a proof of concept for training an SDXL lora on SD1.5 knowledge. Trained on &#39;headshot of &lt;strong&gt;person&lt;/strong&gt;&#39; where &lt;strong&gt;person&lt;/strong&gt; is a wildcard.&lt;br /&gt;&lt;br /&gt;&lt;em&gt;update&lt;/em&gt; much smaller lora + cleaner dataset + trained longer&lt;/p&gt;&lt;p&gt;(&lt;strong&gt;wip&lt;/strong&gt;) Mirror &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://ntcai.xyz/sdxl/headshot&#34;&gt;https://ntcai.xyz/sdxl/headshot&lt;/a&gt;&lt;/p&gt;&lt;p&gt;&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Usage:&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;&lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://ntcai.xyz/comfyui/lorasimplesdxl.json&#34;&gt;https://ntcai.xyz/comfyui/lorasimplesdxl.json&lt;/a&gt; comfyui&lt;/p&gt;&lt;p&gt;&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Quick note:&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;&lt;em&gt;I just setup a patreon at &lt;/em&gt;&lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;http://patreon.com/NTCAI&#34;&gt;&lt;em&gt;patreon.com/NTCAI&lt;/em&gt;&lt;/a&gt;&lt;em&gt; if you&#39;d like to support my work or vote on what to work on next. There&#39;s also an nft drop planned for supporters but it won&#39;t have financial value, just having fun.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;&lt;strong&gt;&lt;br /&gt;This process is:&lt;/strong&gt;&lt;/p&gt;&lt;ol&gt;&lt;li&gt;&lt;p&gt;Use img2img on SDXL output using SD1.5. I wrote a tutorial on this here: &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://civitai.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui&#34;&gt;https://civitai.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui&lt;/a&gt; ( &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://ntcai.xyz/articles/applying_stable_diffusion_15_models_loras_controlnet_to_sdxl_with_comfyui_and_img2img/&#34;&gt;mirror&lt;/a&gt; )&lt;br /&gt;Create a large dataset with this technique - this is trained on 800 images&lt;br /&gt;&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Hint: &lt;/strong&gt;choose a subject matter that sd1.5 knows well and perhaps reject any distorted images.&lt;/p&gt;&lt;/li&gt;&lt;li&gt;&lt;p&gt;Get your files in the correct form. This tutorial helped me:&lt;/p&gt;&lt;div data-youtube-video&gt;&lt;iframe allowfullscreen=&#34;true&#34; autoplay=&#34;false&#34; disablekbcontrols=&#34;false&#34; enableiframeapi=&#34;false&#34; endtime=&#34;0&#34; ivloadpolicy=&#34;0&#34; loop=&#34;false&#34; modestbranding=&#34;false&#34; origin playlist src=&#34;https://www.youtube.com/embed/AY6DMBCIZ3A&#34; start=&#34;0&#34; width=&#34;640&#34; height=&#34;480&#34;&gt;&lt;/iframe&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;p&gt;Train SDXL using &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://github.com/kohya-ss/sd-scripts&#34;&gt;https://github.com/kohya-ss/sd-scripts&lt;/a&gt; on the generated images.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# Full command
CUDA_VISIBLE_DEVICES=0 accelerate launch --num_cpu_threads_per_process=2 &#34;sdxl_train_network.py&#34; --enable_bucket --pretrained_model_name_or_path=&#34;/ml2/trained/ComfyUI/models/checkpoints/stable-diffusion-xl-base-1.0/sd_xl_base_1.0.safetensors&#34; --train_data_dir=&#34;/ml2/trained/sd-scripts/data/headshot&#34; --resolution=&#34;1024,1024&#34; --output_dir=&#34;/ml2/trained/ComfyUI/models/loras/Lora/sdxl&#34; --logging_dir=&#34;./logs&#34; --save_model_as=safetensors --output_name=&#34;headshot2&#34; --network_alpha=&#34;1&#34; --network_dim=32 --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --lr_scheduler=&#34;constant&#34; --train_batch_size=&#34;16&#34; --max_train_steps=&#34;10000&#34; --mixed_precision=&#34;bf16&#34; --save_every_n_epochs=&#34;1&#34; --save_precision=&#34;bf16&#34; --caption_extension=&#34;.txt&#34; --optimizer_type=&#34;Adafactor&#34; --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers=&#34;0&#34; --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&lt;/ol&gt;&lt;p&gt;&lt;br /&gt;My hope is that this can help creators migrate their work. Happy training!&lt;/p&gt;&lt;p&gt;&lt;br /&gt;&lt;br /&gt;&lt;strong&gt;Resources used:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt;&lt;li&gt;&lt;p&gt;eyes lora &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://civitai.com/models/72447/eyes-trained-with-no-data-trick&#34;&gt;https://civitai.com/models/72447/eyes-trained-with-no-data-trick&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&lt;li&gt;&lt;p&gt;SD1.5 aniverse &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://civitai.com/models/107842&#34;&gt;https://civitai.com/models/107842&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&lt;li&gt;&lt;p&gt;person wildcards &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://civitai.com/models/98622/clonecleaner-wildcards-increase-character-consistency&#34;&gt;https://civitai.com/models/98622/clonecleaner-wildcards-increase-character-consistency&lt;/a&gt;&lt;/p&gt;&lt;p&gt;&lt;/p&gt;&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Trained for 8025 steps at batch size 16 on the phrase &#39; headshot of &lt;strong&gt;person&lt;/strong&gt; &#39; where person is a wildcard. The phrase was run through the SDXL to SD1.5 pipeline outlined here &lt;a target=&#34;_blank&#34; rel=&#34;ugc&#34; href=&#34;https://civitai.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui&#34;&gt;https://civitai.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui&lt;/a&gt;&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# Full command
CUDA_VISIBLE_DEVICES=0 accelerate launch --num_cpu_threads_per_process=2 &#34;sdxl_train_network.py&#34; --enable_bucket --pretrained_model_name_or_path=&#34;/ml2/trained/ComfyUI/models/checkpoints/stable-diffusion-xl-base-1.0/sd_xl_base_1.0.safetensors&#34; --train_data_dir=&#34;/ml2/trained/sd-scripts/data/headshot&#34; --resolution=&#34;1024,1024&#34; --output_dir=&#34;/ml2/trained/ComfyUI/models/loras/Lora/sdxl&#34; --logging_dir=&#34;./logs&#34; --save_model_as=safetensors --output_name=&#34;headshot2&#34; --network_alpha=&#34;1&#34; --network_dim=32 --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --lr_scheduler=&#34;constant&#34; --train_batch_size=&#34;16&#34; --max_train_steps=&#34;10000&#34; --mixed_precision=&#34;bf16&#34; --save_every_n_epochs=&#34;1&#34; --save_precision=&#34;bf16&#34; --caption_extension=&#34;.txt&#34; --optimizer_type=&#34;Adafactor&#34; --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers=&#34;0&#34; --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Reducing the network_dim reduced the filesize&lt;/p&gt;

Files with SHA256: e6d28bbc954fcdeedb8872fd726283ab3c03df96b5e14753b0a454505f752e9d