Warning: I just remembered something, the scripts are recursive so you can just drop your images folder inside and it will process them. On the other hand do not just drop it anywhere and run it, if you were to just drop it at c: and run it, it will look for images EVERYWHERE, It won't damage anything but will create a lot of trash. So remember to run it in it's own folder.I posted a <a target="_blank" rel="ugc" href="https://civarchive.com/articles/138">lora making guide</a> i made a while ago. I normally use a couple of powershell scripts to some common tasks. Be mindful some require image magic installed, I dumped the windows installer in the zip file If you don't trust it simply get the original from <a target="_blank" rel="ugc" href="https://github.com/ImageMagick/ImageMagick/releases/">github</a>Included files:<ul><li>90percentsimilar.ps1: Poorman's image duplication finder. First you need to autotag your images. It Checks PNG/TXT file pairs against others in the folder if the tags are over 90% similar they are bundled together in a subfolder</li><li>avifdec.exe: Just the run of the mill avif decoded from github required for topng.ps1</li><li>cleanExtraTxt.ps1: Check IMG/TXT file pairs and moves orphaned text files into a subfolder supports png, jpg, jpeg, bmp, webp, gif and avif</li><li>dwebp.exe: just the run of the mill webp decoder required for topng.ps1</li><li>ffmpeg.exe: run of the mill video decoder it is needed for .MP4 and .gif required for topng.ps1</li><li>gifSplitter.ps1: simply extracts the frames of .gif can use topng.ps1 instead.</li><li>ImageMagick.Q16-HDRI.msixbundle: the image magic windows installer needed for some stuff.</li><li>PNGresizer.ps1: Makes things square. This is no longer useful as bucketing is ubiquitous and works fine.</li><li>PNGresizerToBucket.ps1: This is my other script to validate images needing upscaling, sorting by bucket and some primitive downscaling and or cropping.</li><li>RemoveAlpha.ps1: this one need image magic it does as stated in the can removes and disables the alpha channel or transparency in PNG files. Unless your training script supports transparency it can either screw things up or cause it to fail so unless you are explicitly using transparency training use this to remove it.</li><li>removeBorder.ps1: This one also needs image magic. It checks the borders of PNG files and removes rows and columns from the sides that fall within tolerance limits. I normally use it with 20% tolerance. BEWARE This edits your images so make a copy before running. This is normally very reliable except with anime screencaps from night scenes, somehow it always want to eat those. This one is great to remove white or black borders to get those sweet yet marginal resolution gains when training.</li><li>renamePadnumeric.ps1: this one is a bit of a dumb one, it will pad numeric PNG/TXT files pairs so if they 1.png and 1.txt they will become 000001.png and 000001.txt</li><li>RenamePairs.Ps1: slightly less dumb version of the one above will simply rename PNG/TXT files pairs numerically and in sequence.</li><li>tograyscale.ps1: this one need image magic, as stated in the can will create a grayscale copy of a PNG image. This is useful on already grayscale images. You are probably thinking that's absurd! But nope, a lot of grayscale images are full color and only appear grey, if you zoom in they look like a damn rainbow. So just run it over your grayscale images and use the converted ones if they were already true grayscale nothing will change if they weren't it will prevent the rainbow effect from being trained.</li><li>tomono.ps1: This one needs image magic. This one requires a cut threshold percentage and will turn PNG images into strict monochrome, it is good for lineart that looks grey and washed out. normally works fine between 40 and 60.</li><li>topng.ps1: This will check all the images in a folder and will create their png counterpart with the name "_fromJPEG" or whatever they were it also splits gifs and mp4.</li></ul>Here's how to run it.<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/2c4f9a6f-84c1-4a7b-852b-e750ffac4a2b/width=525/2c4f9a6f-84c1-4a7b-852b-e750ffac4a2b.jpeg" />

Powershell scripts: Misc scripts for dataset operations.

Files with SHA256: 0fb9f7a931408e7123700158dad4ee8b5f77cbc634e8778aa256aa6868b1dc9a