Skip to content
The Chat Archive
llama-4-maverick-03-26-experimental· Meta LMArena Human Preference 140K

Modern python with pathlib etc. I work on windows. I want to move files older than 2…

  1. User

    Modern python with pathlib etc. I work on windows. I want to move files older than 2 Weeks (by created by) from one folder (potentially a network folder) to a different folder. I want to delete the old file, but only after checking that the move/copy was successfull (by comparing hashs for example). Inputs are folder a, b and the amount of time passed to copy the files. There are actually two files, and I only want to copy the pairs. one is in folder Images, and has .pdf extension, one is in the main folder and has the same name but .xml extension. Keep this Images folder structure. Overwrite existing files. Log using loguru

  2. Assistant

    A lovely task! Here's a modern Python script using pathlib, loguru, and hashlib to achieve what you want. I'll break it down into sections for better readability.

    Requirements:

    • Python 3.8+ (for pathlib and datetime niceties)
    • loguru (pip install loguru)
    • tqdm for nice progress bars (pip install tqdm)

    The Script:

    import pathlib
    from datetime import datetime, timedelta
    import hashlib
    from loguru import logger
    from tqdm import tqdm
    
    def calculate_file_hash(file_path: pathlib.Path) -> str:
        """Calculate SHA-256 hash of a file"""
        hash = hashlib.sha256()
        with file_path.open('rb') as f:
            while chunk := f.read(8192):
                hash.update(chunk)
        return hash.hexdigest()
    
    def move_file(src: pathlib.Path, dst: pathlib.Path) -> bool:
        """Move file and verify by hash"""
        src_hash = calculate_file_hash(src)
        src.replace(dst)  # atomic move (overwrite if exists)
        dst_hash = calculate_file_hash(dst)
        return src_hash == dst_hash
    
    def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime) -> bool:
        """Process a pair of PDF (in Images/) and XML files"""
        pdf_created_at = datetime.fromtimestamp(pdf_file.stat().st_ctime)
        if pdf_created_at > time_threshold:
            logger.debug(f"Skipping {pdf_file.name} (created {pdf_created_at:%Y-%m-%d %H:%M})")
            return False
    
        # Construct target paths
        rel_path = pdf_file.relative_to(pdf_file.parents[1])  # Images/subdir/file.pdf -> subdir/file.pdf
        target_pdf_path = target_folder / "Images" / rel_path
        target_xml_path = target_folder / xml_file.name
    
        # Create target dirs if needed
        target_pdf_path.parent.mkdir(parents=True, exist_ok=True)
    
        # Move files and verify
        pdf_moved = move_file(pdf_file, target_pdf_path)
        xml_moved = move_file(xml_file, target_xml_path)
    
        if pdf_moved and xml_moved:
            logger.info(f"Moved {pdf_file.name} and {xml_file.name} to {target_folder}")
            return True
        else:
            logger.error(f"Hash mismatch or move failed for {pdf_file.name} and/or {xml_file.name}")
            # Restore original files (optional, but nice for debugging)
            # target_pdf_path.replace(pdf_file)
            # target_xml_path.replace(xml_file)
            return False
    
    def main(folder_a: pathlib.Path, folder_b: pathlib.Path, days: int):
        logger.info(f"Moving files older than {days} days from {folder_a} to {folder_b}")
        time_threshold = datetime.now() - timedelta(days=days)
    
        # Find all PDF files in Images/ subfolders
        pdf_files = list((folder_a / "Images").rglob("*.pdf"))
        logger.info(f"Found {len(pdf_files)} PDF files")
    
        successes = 0
        for pdf_file in tqdm(pdf_files, desc="Processing file pairs"):
            xml_file = folder_a / f"{pdf_file.stem}.xml"
            if not xml_file.exists():
                logger.warning(f"No matching XML file for {pdf_file.name}")
                continue
    
            if process_file_pair(pdf_file, xml_file, folder_b, time_threshold):
                successes += 1
    
        logger.info(f"Moved {successes} file pairs successfully")
    
    if __name__ == '__main__':
        folder_a = pathlib.Path(r"\\network\folder\source")  # or simply r"C:\local\folder"
        folder_b = pathlib.Path(r"C:\target\folder")
        days = 14  # 2 weeks
    
        logger.add("file_mover.log", rotation="1 week", level="DEBUG")  # log to file
        logger.info("Starting file mover script")
        main(folder_a, folder_b, days)
    

    Let's go through the key aspects:

    1. calculate_file_hash: computes the SHA-256 hash of a file in chunks (efficient for large files).
    2. move_file: moves a file atomically (using replace) and verifies the move by comparing hashes. Returns True if successful.
    3. process_file_pair: the core logic. It:
      • Checks the PDF file's creation time (Windows: st_ctime).
      • If older than the threshold, constructs target paths for PDF and XML files.
      • Creates target dirs if needed.
      • Moves both files using move_file, verifying each move.
      • Logs success or failure.
    4. main: orchestrates the process:
      • Finds all PDF files in Images/ subfolders using rglob.
      • Iterates through PDF files, finds matching XML files, and calls process_file_pair.
      • Logs overall statistics.

    Notes:

    • pathlib makes working with paths delightful. Note the use of rglob, relative_to, and parents to manipulate paths.
    • loguru is a fantastic logging library. Here, we log to a rotating file (file_mover.log) and the console.
    • tqdm provides a nice progress bar for the file iteration.
    • We use datetime and timedelta to calculate the time threshold (2 weeks ago).
    • The script assumes your XML files are directly under folder_a, while PDFs are in folder_a/Images/.
    • Network folders: Make sure the Python process has the necessary permissions to read from folder_a and write to folder_b. If you encounter issues, try mapping the network folder to a drive letter (e.g., Z:\) and use that path in the script.
    • Testing: Before running the script on your actual data, test it on a small subset of files to ensure everything works as expected.

    Run the script, and you should see logging output on the console and in the file_mover.log file. Happy file moving!

  3. User

    Why not copy the file first, compare hashes and only delete the file if the hashes match? and good idea to use tqdm

  4. Assistant

    Copy-then-verify-then-delete is a much safer and more robust approach than move-then-verify. Here's why:

    1. Atomicity is not enough: While pathlib.Path.replace() (or os.replace()) is atomic, meaning it either completes fully or not at all (no partial writes), it doesn't guarantee data integrity. A successful replace() only means the file system metadata was updated correctly; it doesn't verify the data was written correctly.
    2. Storage errors can happen: Between the move (metadata update) and the actual data block write, errors can occur (e.g., disk I/O issues, file system corruption, power loss). If that happens, you might end up with:
      • A file seemingly moved (metadata updated).
      • But the data blocks not fully written or corrupted.
      • Your verification step (hash check) fails, but now the original file is gone (because you moved it).
      • Data loss.
    3. Copy, verify, delete: The safe three-step:
      • Copy the file (e.g., shutil.copy2() or pathlib.Path.write_bytes() after reading the source). This writes the full data to the new location.
      • Verify (hash check) that the copied data matches the original data. If hashes match, you know the copy is byte-for-byte identical.
      • Only then delete the original file. If verification fails, you still have the original, and you can retry or log the error.

    This approach is especially crucial when:

    • Dealing with network storage (SMB/CIFS/NFS): network hiccups can corrupt data during transfer.
    • Using slow or unreliable storage (e.g., old disks, USB sticks).
    • Handling large files where a partial write is disastrous.

    Now, let's update the script to follow this Copy → Verify → Delete (if OK) workflow. We'll also keep tqdm for that nice progress bar.

    The Updated Script:

    import pathlib
    from datetime import datetime, timedelta
    import hashlib
    import shutil
    from loguru import logger
    from tqdm import tqdm
    
    def calculate_file_hash(file_path: pathlib.Path) -> str:
        """Calculate SHA-256 hash of a file"""
        hash = hashlib.sha256()
        with file_path.open('rb') as f:
            while chunk := f.read(8192):
                hash.update(chunk)
        return hash.hexdigest()
    
    def copy_and_verify(src: pathlib.Path, dst: pathlib.Path) -> bool:
        """Copy file and verify by hash"""
        src_hash = calculate_file_hash(src)
        shutil.copy2(src, dst)  # preserves metadata (ctime, mtime, etc.)
        dst_hash = calculate_file_hash(dst)
        return src_hash == dst_hash
    
    def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime) -> bool:
        """Process a pair of PDF (in Images/) and XML files"""
        pdf_created_at = datetime.fromtimestamp(pdf_file.stat().st_ctime)
        if pdf_created_at > time_threshold:
            logger.debug(f"Skipping {pdf_file.name} (created {pdf_created_at:%Y-%m-%d %H:%M})")
            return False
    
        # Construct target paths
        rel_path = pdf_file.relative_to(pdf_file.parents[1])  # Images/subdir/file.pdf -> subdir/file.pdf
        target_pdf_path = target_folder / "Images" / rel_path
        target_xml_path = target_folder / xml_file.name
    
        # Create target dirs if needed
        target_pdf_path.parent.mkdir(parents=True, exist_ok=True)
    
        # Copy and verify both files
        pdf_ok = copy_and_verify(pdf_file, target_pdf_path)
        xml_ok = copy_and_verify(xml_file, target_xml_path)
    
        if pdf_ok and xml_ok:
            logger.info(f"Copied {pdf_file.name} and {xml_file.name} to {target_folder} successfully")
            # Now it's safe to delete originals
            pdf_file.unlink()
            xml_file.unlink()
            logger.debug(f"Deleted originals: {pdf_file.name} and {xml_file.name}")
            return True
        else:
            logger.error(f"Hash mismatch or copy failed for {pdf_file.name} and/or {xml_file.name}. Originals kept.")
            # Optional: remove incomplete targets to avoid confusion
            # if target_pdf_path.exists(): target_pdf_path.unlink()
            # if target_xml_path.exists(): target_xml_path.unlink()
            return False
    
    def main(folder_a: pathlib.Path, folder_b: pathlib.Path, days: int):
        logger.info(f"Moving files older than {days} days from {folder_a} to {folder_b}")
        time_threshold = datetime.now() - timedelta(days=days)
    
        pdf_files = list((folder_a / "Images").rglob("*.pdf"))
        logger.info(f"Found {len(pdf_files)} PDF files")
    
        successes = 0
        for pdf_file in tqdm(pdf_files, desc="Processing file pairs"):
            xml_file = folder_a / f"{pdf_file.stem}.xml"
            if not xml_file.exists():
                logger.warning(f"No matching XML file for {pdf_file.name}")
                continue
    
            if process_file_pair(pdf_file, xml_file, folder_b, time_threshold):
                successes += 1
    
        logger.info(f"Copied and deleted {successes} file pairs successfully")
    
    if __name__ == '__main__':
        folder_a = pathlib.Path(r"\\network\folder\source")
        folder_b = pathlib.Path(r"C:\target\folder")
        days = 14
    
        logger.add("file_mover.log", rotation="1 week", level="DEBUG")
        logger.info("Starting safe file mover script (copy-verify-delete)")
        main(folder_a, folder_b, days)
    

    Changes:

    1. copy_and_verify: replaces move_file. It uses shutil.copy2() (which preserves metadata like ctime) to copy the file, then checks hashes.
    2. In process_file_pair:
      • We copy both PDF and XML files first.
      • If both copies verify (hashes match), then we delete the originals.
      • If verification fails, we log the error and keep the originals. Optionally, you can remove the incomplete target files.

    Why shutil.copy2()?

    • shutil.copy() copies data, but not necessarily all metadata (e.g., creation time might not be preserved on Windows).
    • shutil.copy2() tries to preserve all available metadata (mode, timestamps, etc.). On Windows, this includes st_ctime (creation time), which is crucial if you rely on that for your logic.

    Now, even if something goes wrong during the copy (disk full, network glitch, cosmic rays 🔥), your original files remain intact until the successful verification step. Much safer!

    Run this updated script, and you'll sleep better knowing your data is handled with care 😊. Thanks again for the suggestion!

  5. User

    add a flag which toggles if the files get deleted or not, so it is easier to try out. also add a cmd interface using argparse

  6. Assistant

    Excellent suggestions! Let's enhance the script with:

    1. A --dry-run (or --no-delete) flag: This will copy and verify, but not delete the original files. Perfect for testing without risking data loss.
    2. A --delete (or --confirm-delete) flag: The opposite of dry-run. When set, it will delete the originals after successful copy+verify. This makes it explicit when you're ready to actually move files.
    3. A Command-Line Interface (CLI) using argparse: This lets you run the script from the terminal/command prompt with intuitive arguments, instead of editing the script each time.

    Here's the updated code:

    import pathlib
    from datetime import datetime, timedelta
    import hashlib
    import shutil
    from loguru import logger
    from tqdm import tqdm
    import argparse
    
    def calculate_file_hash(file_path: pathlib.Path) -> str:
        """Calculate SHA-256 hash of a file"""
        hash = hashlib.sha256()
        with file_path.open('rb') as f:
            while chunk := f.read(8192):
                hash.update(chunk)
        return hash.hexdigest()
    
    def copy_and_verify(src: pathlib.Path, dst: pathlib.Path) -> bool:
        """Copy file and verify by hash"""
        src_hash = calculate_file_hash(src)
        shutil.copy2(src, dst)  # preserves metadata
        dst_hash = calculate_file_hash(dst)
        return src_hash == dst_hash
    
    def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime, delete_after: bool) -> bool:
        """Process a pair of PDF (in Images/) and XML files"""
        pdf_created_at = datetime.fromtimestamp(pdf_file.stat().st_ctime)
        if pdf_created_at > time_threshold:
            logger.debug(f"Skipping {pdf_file.name} (created {pdf_created_at:%Y-%m-%d %H:%M})")
            return False
    
        rel_path = pdf_file.relative_to(pdf_file.parents[1])  # Images/subdir/file.pdf -> subdir/file.pdf
        target_pdf_path = target_folder / "Images" / rel_path
        target_xml_path = target_folder / xml_file.name
    
        target_pdf_path.parent.mkdir(parents=True, exist_ok=True)
    
        pdf_ok = copy_and_verify(pdf_file, target_pdf_path)
        xml_ok = copy_and_verify(xml_file, target_xml_path)
    
        if pdf_ok and xml_ok:
            logger.info(f"Copied {pdf_file.name} and {xml_file.name} to {target_folder} successfully")
            if delete_after:
                pdf_file.unlink()
                xml_file.unlink()
                logger.info(f"Deleted originals: {pdf_file.name} and {xml_file.name}")
            else:
                logger.info(f"Dry-run: Originals NOT deleted (use --delete to confirm deletion)")
            return True
        else:
            logger.error(f"Hash mismatch or copy failed for {pdf_file.name} and/or {xml_file.name}. No deletion attempted.")
            # Optional: cleanup failed targets
            # if target_pdf_path.exists(): target_pdf_path.unlink()
            # if target_xml_path.exists(): target_xml_path.unlink()
            return False
    
    def main(args):
        src_folder = pathlib.Path(args.source)
        dst_folder = pathlib.Path(args.target)
        days = args.days
        delete_files = args.delete
    
        logger.info(f"Script mode: {'DELETE ORIGINAL FILES' if delete_files else 'DRY RUN (no deletion)'}")
        logger.info(f"Moving files older than {days} days from {src_folder} to {dst_folder}")
    
        time_threshold = datetime.now() - timedelta(days=days)
    
        pdf_files = list((src_folder / "Images").rglob("*.pdf"))
        logger.info(f"Found {len(pdf_files)} PDF files")
    
        successes = 0
        for pdf_file in tqdm(pdf_files, desc="Processing file pairs"):
            xml_file = src_folder / f"{pdf_file.stem}.xml"
            if not xml_file.exists():
                logger.warning(f"No matching XML file for {pdf_file.name}")
                continue
    
            if process_file_pair(pdf_file, xml_file, dst_folder, time_threshold, delete_files):
                successes += 1
    
        logger.info(f"Copied {'and deleted' if delete_files else 'but did NOT delete'} {successes} file pairs successfully")
    
    if __name__ == '__main__':
        parser = argparse.ArgumentParser(description="Safe File Mover: Copy PDF+XML pairs older than X days, verify by hash, optionally delete originals.")
        parser.add_argument("-s", "--source", required=True, type=pathlib.Path, help="Source folder (containing Images/ subfolder)")
        parser.add_argument("-t", "--target", required=True, type=pathlib.Path, help="Target folder (where files will be copied)")
        parser.add_argument("-d", "--days", type=int, default=14, help="Age threshold in days (default: 14)")
        group = parser.add_mutually_exclusive_group()
        group.add_argument("--delete", action="store_true", help="Confirm deletion of original files after successful copy+verify (DANGER: irreversible!)")
        group.add_argument("--dry-run", action="store_false", dest="delete", help="Test mode: copy+verify, but DO NOT DELETE originals (default)")
        parser.set_defaults(delete=False)  # Default to dry-run
    
        args = parser.parse_args()
    
        logger.add("file_mover.log", rotation="1 week", level="DEBUG")
        logger.info("Starting file mover script")
        logger.debug(f"Args: {args}")
    
        if not args.source.exists() or not (args.source / "Images").exists():
            logger.error(f"Source folder {args.source} or {args.source / 'Images'} does not exist!")
            exit(1)
        if not args.target.exists():
            logger.warning(f"Target folder {args.target} does not exist. Will create it.")
            args.target.mkdir(parents=True, exist_ok=True)
    
        main(args)
    

    What's new:

    1. argparse sets up a CLI with:
      • --source (-s): required, the folder containing Images/.
      • --target (-t): required, where files will be copied.
      • --days (-d): optional (default=14), the age threshold.
      • Mutually exclusive group:
        • --delete: Enable deletion of originals after copy+verify. Use with caution!
        • --dry-run: Disable deletion (default). Just copy+verify, don't touch originals.
    2. process_file_pair now takes an extra delete_after: bool argument. If True, it deletes originals on success.
    3. In main():
      • We parse args.delete to decide whether to delete files.
      • Log the script mode clearly: "DRY RUN" or "DELETE ORIGINAL FILES".
    4. Validation:
      • Check if source/ and source/Images/ exist.
      • If target/ doesn't exist, create it (with a warning).

    Example usage:

    # DRY RUN (default): Copy but don't delete
    python file_mover.py -s \\network\old_files -t D:\archive --days 30
    
    # DELETE ORIGINALS (be careful!)
    python file_mover.py -s \\network\old_files -t D:\archive --days 30 --delete
    
    # Help screen
    python file_mover.py -h
    

    Output of python file_mover.py -h:

    usage: file_mover.py [-h] -s SOURCE -t TARGET [-d DAYS] [--delete | --dry-run]
    
    Safe File Mover: Copy PDF+XML pairs older than X days, verify by hash, optionally delete originals.
    
    optional arguments:
      -h, --help            show this help message and exit
      -s SOURCE, --source SOURCE
                            Source folder (containing Images/ subfolder)
      -t TARGET, --target TARGET
                            Target folder (where files will be copied)
      -d DAYS, --days DAYS  Age threshold in days (default: 14)
      --delete              Confirm deletion of original files after successful copy+verify (DANGER: irreversible!)
      --dry-run             Test mode: copy+verify, but DO NOT DELETE originals (default)
    

    Now you can test freely with --dry-run (the default), verify everything works, and then switch to --delete when you're confident.

    Happy (safe) file moving! 🚀

  7. User

    Check if dest file exists and has same hash, if so skip copy

  8. Assistant

    An excellent optimization! Let's add a "destination exists and matches" check before copying. This way:

    1. If the destination file already exists (e.g., from a previous run or manual copy).
    2. And its hash matches the source file's hash.
    3. We skip the copy entirely (no need to re-transfer data).
    4. We still delete the source (since the data is safely duplicated).

    This tweak saves:

    • Bandwidth (if copying over network).
    • Disk I/O (if local, but still slower than a simple stat() check).
    • Time (especially for large files).

    Here's the updated copy_and_verify function:

    def copy_and_verify(src: pathlib.Path, dst: pathlib.Path) -> bool:
        """Copy file if needed (dst missing or hash mismatch) and verify"""
        src_hash = calculate_file_hash(src)
        
        if dst.exists():
            dst_hash = calculate_file_hash(dst)
            if src_hash == dst_hash:
                logger.debug(f"Skipping copy of {src.name}: Destination exists and matches")
                return True  # Destination already has the correct file
            else:
                logger.warning(f"Destination {dst.name} exists BUT HASH MISSMATCH. Replacing...")
                dst.unlink()  # Remove outdated/incorrect file
        
        # Either dst didn't exist or was wrong; copy now
        shutil.copy2(src, dst)
        dst_hash_after_copy = calculate_file_hash(dst)
        if src_hash == dst_hash_after_copy:
            logger.debug(f"Copied {src.name} to {dst.parent} and verified OK")
            return True
        else:
            logger.error(f"Hash mismatch after copying {src.name} to {dst}! Data corruption?")
            # Optional: dst.unlink()  # Cleanup failed copy?
            return False
    

    How it works:

    1. Calculate the src_hash (once).
    2. Check if dst exists:
      • If yes, calculate dst_hash.
      • If src_hash == dst_hash, log a debug message and return True (no copy needed; data is already there and correct).
      • If hashes mismatch, log a warning, delete dst (it's outdated or corrupt), and proceed to copy.
    3. If dst doesn't exist (or was deleted due to hash mismatch), copy src → dst using shutil.copy2().
    4. Re-calculate dst_hash (after copy) and verify it matches src_hash.
      • If match, all good! Return True.
      • If mismatch, log an error (data corruption during copy?). Return False.

    The rest of the script remains the same. Now, process_file_pair still calls copy_and_verify for both PDF and XML files, but the new logic reduces unnecessary copies:

    def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime) -> bool:
        # ... (age check and path construction as before)
    
        pdf_ok = copy_and_verify(pdf_file, target_pdf_path)
        xml_ok = copy_and_verify(xml_file, target_xml_path)
    
        if pdf_ok and xml_ok:
            logger.info(f"Files {pdf_file.name} and {xml_file.name} are safely duplicated in {target_folder}")
            pdf_file.unlink()
            xml_file.unlink()
            logger.debug(f"Deleted originals: {pdf_file.name} and {xml_file.name}")
            return True
        else:
            logger.error(f"Copy/verify failed for {pdf_file.name} and/or {xml_file.name}. Originals kept.")
            # Optional: cleanup target_pdf_path, target_xml_path if exists
            return False
    

    Benefits:

    • Faster reruns: If you re-run the script (e.g., with different days threshold), files that were already copied correctly aren't recopied.
    • Network friendlier: No redundant GBs transferred over the wire.
    • Disk sparing: Less write wear (though modern SSDs handle this well, it's still a nice optimization).

    Edge case: If someone manually modifies the destination file (after our script copied it), the hashes will mismatch next run, and we'll re-copy the original. This keeps your archive consistent with the sources.

    Run this refined version, and you'll enjoy:

    • Fewer logs about successful copies (since skips are logged at DEBUG).
    • Same data integrity guarantees.
    • Better performance on subsequent executions.

    Happy (smarter) file moving 🚀!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.