Image Corruption and Text Validation Script

What the Script Does

This script scans a specified directory containing images and text files to perform the following tasks:

1. Detects Corrupted Images:

   • Images in formats like .png, .jpg, .jpeg, .gif, .bmp are checked for corruption.

   • If corrupted, the image is moved to an output directory specified in the config.yaml file.

2. Detects Empty Text Files:

   • Scans all .txt files and checks if they are empty.

   • Logs the paths of empty text files.

3. Finds Images Without Corresponding Text Files:

   • Checks if an image (e.g., example.png) has a corresponding text file (e.g., example.txt).

   • Logs any images missing a corresponding .txt file.

4. Warns About WEBP Images:

   • Detects .webp images and logs a warning.

   • Advises users to convert .webp files to .png or .jpg as they are unsuitable for training.

5. Logs Results:

   • Generates a log file (scan_log_<date>.txt) in the output directory.

   • The log includes:

     • A list of corrupted images.

     • Empty text files.

     • Images without corresponding text files.

     • .webp files that need conversion.

How to Use the Script

1. Install Required Libraries:

Run the following command to install dependencies:

pip install -r requirements.txt

Prepare the Configuration File (config.yaml):

Create a config.yaml file in the same directory as the script with the following structure:

image_directory: "path/to/your/images"  # Directory containing your images and text files
output_directory: "path/to/output/directory"  # Directory where corrupted images will be moved, and logs will be saved

Replace path/to/your/images and path/to/output/directory with actual directory paths.
Run the Script:

Execute the script in the terminal or command prompt:

python check_images.py

1. Check the Output:

   • The script will scan the specified directories.

   • Logs will be saved in the output directory with a name like scan_log_<date>.txt.

2. Review the Log File:

   • Open the log file to see:

     • Corrupted images (moved to the output directory).

     • Empty text files.

     • Images missing text files.

     • .webp images requiring conversion.

Example

If your config.yaml file contains:

image_directory: "C:/Users/user/images"
output_directory: "C:/Users/user/output"

• The script will scan all images and text files in C:/Users/user/images.

• Corrupted images will be moved to C:/Users/user/output.

The log file will also be saved in C:/Users/user/output.

DEEP SCAN SCRIPT
Explanation of the Script

This script is designed to validate and clean image datasets. It scans through folders containing images and text files, ensuring the data is ready for training or other workflows by identifying potential issues.

Features of the Script:

1. Image Validation:

   • The script checks each image file to ensure it is not corrupted.

   • Valid images are logged as "Valid" in the log file.

   • Corrupted images are:

     • Logged as "Corrupted" with the reason.

     • Automatically moved to a specified output folder for manual review.

2. Text File Validation:

   • It checks if .txt files exist without corresponding images in the folder.

   • These "orphaned" text files are logged for user review.

3. Batch Processing:

   • Processes files in small batches to handle large datasets efficiently without overloading the system.

4. Multiprocessing:

   • Uses multiple CPU cores to speed up the validation process, making it faster for large datasets.

5. Real-Time Logging:

   • Creates a log file (corrupted_images_log.txt) that is updated in real-time as the script processes each file.

   • The log includes:

     • Valid images.

     • Corrupted images and the reason for corruption.

     • Orphaned text files.

6. Output Folder for Corrupted Files:

   • Corrupted images are automatically moved to a dedicated output folder, making it easier to locate and fix them.