Skip to content
TheChatArchiveThe Chat Archive
Metallama-4-maverick-17b-128e-instruct· MetaHugging Face LMArena Human Preference 140K

Modern python with pathlib etc. I work on windows. I want to move files older than 2…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Metallama-4-maverick-03-26-experimental’s answer.
  1. User

    Modern python with pathlib etc. I work on windows. I want to move files older than 2 Weeks (by created by) from one folder (potentially a network folder) to a different folder. I want to delete the old file, but only after checking that the move/copy was successfull (by comparing hashs for example). Inputs are folder a, b and the amount of time passed to copy the files. There are actually two files, and I only want to copy the pairs. one is in folder Images, and has .pdf extension, one is in the main folder and has the same name but .xml extension. Keep this Images folder structure. Overwrite existing files. Log using loguru

  2. Meta Assistant

    To accomplish the task you described, we'll use Python with pathlib for handling paths, loguru for logging, hashlib for calculating file hashes, and shutil for moving files. The script will be designed to work on Windows and will be able to handle network folders.

    First, ensure you have the necessary packages installed. You can install them using pip if you haven't already:

    pip install loguru
    

    Here's a Python script that meets your requirements:

    import pathlib
    import loguru
    from datetime import datetime, timedelta
    import hashlib
    import shutil
    
    # Initialize logger
    logger = loguru.logger
    
    def calculate_file_hash(file_path):
        """Calculates the SHA-256 hash of a file."""
        sha256_hash = hashlib.sha256()
        with open(file_path, "rb") as f:
            # Read and update hash in chunks of 4K
            for byte_block in iter(lambda: f.read(4096), b""):
                sha256_hash.update(byte_block)
        return sha256_hash.hexdigest()
    
    def move_file_if_valid(source, destination):
        """Moves a file and checks if the move was successful by comparing hashes."""
        try:
            source_hash = calculate_file_hash(source)
            shutil.move(str(source), str(destination))
            destination_hash = calculate_file_hash(destination)
            if source_hash == destination_hash:
                logger.info(f"Successfully moved {source} to {destination}")
                return True
            else:
                logger.error(f"Hash mismatch after moving {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}")
                # Attempt to recover or handle the error
                return False
        except Exception as e:
            logger.error(f"Error moving {source} to {destination}: {e}")
            return False
    
    def main(folder_a, folder_b, days_threshold):
        folder_a = pathlib.Path(folder_a)
        folder_b = pathlib.Path(folder_b)
        images_folder = folder_a / 'Images'
        
        cutoff_date = datetime.now() - timedelta(days=days_threshold)
        
        for xml_file in folder_a.glob('*.xml'):
            pdf_file = images_folder / (xml_file.stem + '.pdf')
            if pdf_file.exists():
                xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime)
                if xml_created_time < cutoff_date:
                    relative_path = pdf_file.relative_to(images_folder)
                    destination_pdf = folder_b / 'Images' / relative_path
                    destination_xml = folder_b / xml_file.name
                    
                    # Ensure destination directories exist
                    destination_pdf.parent.mkdir(parents=True, exist_ok=True)
                    destination_xml.parent.mkdir(parents=True, exist_ok=True)
                    
                    # Move files
                    move_file_if_valid(pdf_file, destination_pdf)
                    move_file_if_valid(xml_file, destination_xml)
            else:
                logger.warning(f"No corresponding PDF found for {xml_file}")
    
    if __name__ == "__main__":
        folder_a = r"\\path\to\source\folder"  # Update this path
        folder_b = r"\\path\to\destination\folder"  # Update this path
        days_threshold = 14  # 2 weeks
        
        main(folder_a, folder_b, days_threshold)
    

    Key Points:

    • The script uses pathlib for path manipulations, making it easy to work with Windows paths and network locations.
    • It checks for the existence of corresponding .pdf and .xml files and only moves them if both exist.
    • The move_file_if_valid function moves a file and verifies the move by comparing the source and destination file hashes.
    • The script logs its actions and any errors encountered using loguru.
    • It preserves the structure of the 'Images' folder when moving files.
    • You can adjust folder_a, folder_b, and days_threshold in the if __name__ == "__main__": block to suit your needs.

    Before Running:

    1. Replace \\path\to\source\folder and \\path\to\destination\folder with your actual source and destination folder paths.
    2. Ensure the Python script has the necessary permissions to read from the source folder and write to the destination folder.

    This script should be a good starting point for your task. You might need to tweak it based on your specific requirements or environment.

  3. User

    Why not copy the file first, compare hashes and only delete the file if the hashes match? and good idea to use tqdm

  4. Meta Assistant

    Copying the file first and then comparing hashes is a much safer approach than moving the file and then checking the hash. If the hash comparison fails after moving, you've already lost the original file. By copying first, you ensure that the original data is preserved until you've verified the copy.

    Here's an updated version of the script that incorporates this approach and also adds tqdm for progress indication:

    import pathlib
    import loguru
    from datetime import datetime, timedelta
    import hashlib
    import shutil
    from tqdm import tqdm
    
    # Initialize logger
    logger = loguru.logger
    
    def calculate_file_hash(file_path):
        """Calculates the SHA-256 hash of a file."""
        sha256_hash = hashlib.sha256()
        with open(file_path, "rb") as f:
            # Read and update hash in chunks of 4K
            for byte_block in iter(lambda: f.read(4096), b""):
                sha256_hash.update(byte_block)
        return sha256_hash.hexdigest()
    
    def copy_and_move_file_if_valid(source, destination):
        """Copies a file, checks if the copy was successful by comparing hashes, and then removes the original if successful."""
        try:
            source_hash = calculate_file_hash(source)
            shutil.copy2(str(source), str(destination))  # Copy with metadata
            destination_hash = calculate_file_hash(destination)
            if source_hash == destination_hash:
                logger.info(f"Successfully copied {source} to {destination}")
                source.unlink()  # Remove the original file
                logger.info(f"Removed original file {source}")
            else:
                logger.error(f"Hash mismatch after copying {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}")
                destination.unlink()  # Remove the failed copy
        except Exception as e:
            logger.error(f"Error copying {source} to {destination}: {e}")
    
    def main(folder_a, folder_b, days_threshold):
        folder_a = pathlib.Path(folder_a)
        folder_b = pathlib.Path(folder_b)
        images_folder = folder_a / 'Images'
        
        cutoff_date = datetime.now() - timedelta(days=days_threshold)
        
        xml_files = list(folder_a.glob('*.xml'))
        logger.info(f"Found {len(xml_files)} XML files in {folder_a}")
        
        for xml_file in tqdm(xml_files, desc="Processing XML files"):
            pdf_file = images_folder / (xml_file.stem + '.pdf')
            if pdf_file.exists():
                xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime)
                if xml_created_time < cutoff_date:
                    relative_path = pdf_file.relative_to(images_folder)
                    destination_pdf = folder_b / 'Images' / relative_path
                    destination_xml = folder_b / xml_file.name
                    
                    # Ensure destination directories exist
                    destination_pdf.parent.mkdir(parents=True, exist_ok=True)
                    destination_xml.parent.mkdir(parents=True, exist_ok=True)
                    
                    # Copy and potentially remove original files
                    copy_and_move_file_if_valid(pdf_file, destination_pdf)
                    copy_and_move_file_if_valid(xml_file, destination_xml)
            else:
                logger.warning(f"No corresponding PDF found for {xml_file}")
    
    if __name__ == "__main__":
        folder_a = r"\\path\to\source\folder"  # Update this path
        folder_b = r"\\path\to\destination\folder"  # Update this path
        days_threshold = 14  # 2 weeks
        
        main(folder_a, folder_b, days_threshold)
    

    Changes:

    • The copy_and_move_file_if_valid function now copies the file first, compares hashes, and only removes the original file if the hashes match.
    • tqdm is used to display a progress bar when processing XML files.
    • The script logs the number of XML files found at the start.

    This updated approach ensures data integrity by verifying the copy before removing the original.

  5. User

    add a flag which toggles if the files get deleted or not, so it is easier to try out. also add a cmd interface using argparse

  6. Meta Assistant

    Here's the updated script with a flag to toggle file deletion and a command-line interface using argparse:

    import pathlib
    import loguru
    from datetime import datetime, timedelta
    import hashlib
    import shutil
    from tqdm import tqdm
    import argparse
    
    # Initialize logger
    logger = loguru.logger
    
    def calculate_file_hash(file_path):
        """Calculates the SHA-256 hash of a file."""
        sha256_hash = hashlib.sha256()
        with open(file_path, "rb") as f:
            # Read and update hash in chunks of 4K
            for byte_block in iter(lambda: f.read(4096), b""):
                sha256_hash.update(byte_block)
        return sha256_hash.hexdigest()
    
    def copy_and_move_file_if_valid(source, destination, delete_original):
        """Copies a file, checks if the copy was successful by comparing hashes, and then removes the original if successful and delete_original is True."""
        try:
            source_hash = calculate_file_hash(source)
            shutil.copy2(str(source), str(destination))  # Copy with metadata
            destination_hash = calculate_file_hash(destination)
            if source_hash == destination_hash:
                logger.info(f"Successfully copied {source} to {destination}")
                if delete_original:
                    source.unlink()  # Remove the original file
                    logger.info(f"Removed original file {source}")
                else:
                    logger.info(f"Skipping removal of original file {source} due to --dry-run")
            else:
                logger.error(f"Hash mismatch after copying {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}")
                destination.unlink()  # Remove the failed copy
        except Exception as e:
            logger.error(f"Error copying {source} to {destination}: {e}")
    
    def main(folder_a, folder_b, days_threshold, delete_original):
        folder_a = pathlib.Path(folder_a)
        folder_b = pathlib.Path(folder_b)
        images_folder = folder_a / 'Images'
        
        cutoff_date = datetime.now() - timedelta(days=days_threshold)
        
        xml_files = list(folder_a.glob('*.xml'))
        logger.info(f"Found {len(xml_files)} XML files in {folder_a}")
        
        for xml_file in tqdm(xml_files, desc="Processing XML files"):
            pdf_file = images_folder / (xml_file.stem + '.pdf')
            if pdf_file.exists():
                xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime)
                if xml_created_time < cutoff_date:
                    relative_path = pdf_file.relative_to(images_folder)
                    destination_pdf = folder_b / 'Images' / relative_path
                    destination_xml = folder_b / xml_file.name
                    
                    # Ensure destination directories exist
                    destination_pdf.parent.mkdir(parents=True, exist_ok=True)
                    destination_xml.parent.mkdir(parents=True, exist_ok=True)
                    
                    # Copy and potentially remove original files
                    copy_and_move_file_if_valid(pdf_file, destination_pdf, delete_original)
                    copy_and_move_file_if_valid(xml_file, destination_xml, delete_original)
            else:
                logger.warning(f"No corresponding PDF found for {xml_file}")
    
    if __name__ == "__main__":
        parser = argparse.ArgumentParser(description="Copy and optionally remove old XML and PDF files")
        parser.add_argument("-a", "--source", required=True, help="Source folder path")
        parser.add_argument("-b", "--destination", required=True, help="Destination folder path")
        parser.add_argument("-d", "--days", type=int, required=True, help="Number of days to consider a file old")
        parser.add_argument("--dry-run", action="store_false", dest="delete_original", help="Copy files but do not remove originals (default)")
        parser.add_argument("--delete", action="store_true", dest="delete_original", help="Copy files and remove originals")
        parser.add_argument("-v", "--verbose", action="store_true", help="Increase verbosity (not implemented)")
        args = parser.parse_args()
        
        if args.delete_original:
            logger.info("Files will be deleted after successful copy")
        else:
            logger.info("Files will not be deleted after copy (dry run)")
        
        main(args.source, args.destination, args.days, args.delete_original)
    

    Changes:

    • Added a --dry-run flag (default) and a --delete flag to toggle the deletion of original files after copying.
    • Implemented a command-line interface using argparse to accept source folder, destination folder, number of days, and the delete flag as arguments.
    • The script now logs whether it will delete files or not based on the --delete or --dry-run flag.

    You can run the script from the command line like this:

    python script_name.py -a /path/to/source -b /path/to/destination -d 14 --delete
    

    Or for a dry run:

    python script_name.py -a /path/to/source -b /path/to/destination -d 14
    
  7. User

    Check if dest file exists and has same hash, if so skip copy

  8. Meta Assistant

    Here's an updated version of the script that checks if the destination file exists and has the same hash before copying:

    import pathlib
    import loguru
    from datetime import datetime, timedelta
    import hashlib
    import shutil
    from tqdm import tqdm
    
    # Initialize logger
    logger = loguru.logger
    
    def calculate_file_hash(file_path):
        """Calculates the SHA-256 hash of a file."""
        sha256_hash = hashlib.sha256()
        with open(file_path, "rb") as f:
            # Read and update hash in chunks of 4K
            for byte_block in iter(lambda: f.read(4096), b""):
                sha256_hash.update(byte_block)
        return sha256_hash.hexdigest()
    
    def copy_and_move_file_if_valid(source, destination, delete_original):
        """Copies a file, checks if the copy was successful by comparing hashes, and then removes the original if successful and delete_original is True."""
        try:
            if destination.exists():
                source_hash = calculate_file_hash(source)
                destination_hash = calculate_file_hash(destination)
                if source_hash == destination_hash:
                    logger.info(f"Skipping copy of {source} to {destination} as it already exists with the same hash")
                    if delete_original:
                        source.unlink()  # Remove the original file
                        logger.info(f"Removed original file {source}")
                    return
                else:
                    logger.info(f"Destination file {destination} exists but has a different hash. Overwriting...")
            
            source_hash = calculate_file_hash(source)
            shutil.copy2(str(source), str(destination))  # Copy with metadata
            destination_hash = calculate_file_hash(destination)
            if source_hash == destination_hash:
                logger.info(f"Successfully copied {source} to {destination}")
                if delete_original:
                    source.unlink()  # Remove the original file
                    logger.info(f"Removed original file {source}")
            else:
                logger.error(f"Hash mismatch after copying {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}")
                destination.unlink()  # Remove the failed copy
        except Exception as e:
            logger.error(f"Error copying {source} to {destination}: {e}")
    
    def main(folder_a, folder_b, days_threshold, delete_original):
        folder_a = pathlib.Path(folder_a)
        folder_b = pathlib.Path(folder_b)
        images_folder = folder_a / 'Images'
        
        cutoff_date = datetime.now() - timedelta(days=days_threshold)
        
        xml_files = list(folder_a.glob('*.xml'))
        logger.info(f"Found {len(xml_files)} XML files in {folder_a}")
        
        for xml_file in tqdm(xml_files, desc="Processing XML files"):
            pdf_file = images_folder / (xml_file.stem + '.pdf')
            if pdf_file.exists():
                xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime)
                if xml_created_time < cutoff_date:
                    relative_path = pdf_file.relative_to(images_folder)
                    destination_pdf = folder_b / 'Images' / relative_path
                    destination_xml = folder_b / xml_file.name
                    
                    # Ensure destination directories exist
                    destination_pdf.parent.mkdir(parents=True, exist_ok=True)
                    destination_xml.parent.mkdir(parents=True, exist_ok=True)
                    
                    # Copy and potentially remove original files
                    copy_and_move_file_if_valid(pdf_file, destination_pdf, delete_original)
                    copy_and_move_file_if_valid(xml_file, destination_xml, delete_original)
            else:
                logger.warning(f"No corresponding PDF found for {xml_file}")
    
    if __name__ == "__main__":
        folder_a = r"\\path\to\source\folder"  # Update this path
        folder_b = r"\\path\to\destination\folder"  # Update this path
        days_threshold = 14  # 2 weeks
        delete_original = True  # Set to False for a dry run
        
        main(folder_a, folder_b, days_threshold, delete_original)
    

    Changes:

    • The copy_and_move_file_if_valid function now checks if the destination file exists and has the same hash as the source file. If so, it skips the copy operation.
    • The delete_original parameter allows you to control whether the original files are deleted after a successful copy. Set it to False for a dry run.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.