Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

Write Python script which copy png images from "input/" dir to "output/" dir with…

  1. User

    Write Python script which copy png images from "input/" dir to "output/" dir with removing all duplicates

  2. Assistant

    To copy .png images from one directory to another while removing duplicates, you'll need to compare the contents of the images to ensure that no visually identical files are copied over. You can use hashing to accomplish this. Below is a Python script that uses the PIL library from Pillow for image handling and hashlib for hashing:

    import os
    from PIL import Image
    import hashlib
    import shutil
    
    # Define the input and output directories
    input_dir = 'input/'
    output_dir = 'output/'
    
    # Create the output directory if it does not exist
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)
    
    # Helper function to calculate the hash of an image file
    def get_image_hash(img_path):
        with Image.open(img_path) as img:
            # Some images may have an alpha layer, convert to RGB to avoid issues
            image_data = img.convert("RGB").tobytes()
            # Create a hash of the image data
            return hashlib.md5(image_data).hexdigest()
    
    # Keep track of image hashes to identify duplicates
    image_hashes = set()
    
    for file_name in os.listdir(input_dir):
        if file_name.lower().endswith('.png'):
            # Build the full file path
            file_path = os.path.join(input_dir, file_name)
    
            # Calculate the image hash
            img_hash = get_image_hash(file_path)
    
            # If the hash is new (non-duplicate), copy the image
            if img_hash not in image_hashes:
                image_hashes.add(img_hash)
                dest_path = os.path.join(output_dir, file_name)
                shutil.copy(file_path, dest_path)
                print(f"Copied: {file_name}")
            else:
                print(f"Duplicate found, skipping: {file_name}")
    
    print("Finished copying images.")
    

    This script uses MD5 hash which should be sufficient for detecting duplicates among .png files. However, MD5 is not collision-proof for use in security-related applications. There might still be a tiny chance of collision, but when used for duplicate detection in a set of images, the likelihood of encountering a duplicate is extremely low.

    Make sure to install Pillow if you haven't already installed it before you run the script:

    pip install pillow
    

    Run the script from the same directory as the input/ folder, and it will process the PNG files, checking for duplicates by their content, and copy unique images to the output/ folder.