Modern python with pathlib etc. I work on windows. I want to move files older than 2…
- User
Modern python with pathlib etc. I work on windows. I want to move files older than 2 Weeks (by created by) from one folder (potentially a network folder) to a different folder. I want to delete the old file, but only after checking that the move/copy was successfull (by comparing hashs for example). Inputs are folder a, b and the amount of time passed to copy the files. There are actually two files, and I only want to copy the pairs. one is in folder Images, and has .pdf extension, one is in the main folder and has the same name but .xml extension. Keep this Images folder structure. Overwrite existing files. Log using loguru
- Assistant
A lovely task! Here's a modern Python script using
pathlib,loguru, andhashlibto achieve what you want. I'll break it down into sections for better readability.Requirements:
- Python 3.8+ (for
pathlibanddatetimeniceties) loguru(pip install loguru)tqdmfor nice progress bars (pip install tqdm)
The Script:
import pathlib from datetime import datetime, timedelta import hashlib from loguru import logger from tqdm import tqdm def calculate_file_hash(file_path: pathlib.Path) -> str: """Calculate SHA-256 hash of a file""" hash = hashlib.sha256() with file_path.open('rb') as f: while chunk := f.read(8192): hash.update(chunk) return hash.hexdigest() def move_file(src: pathlib.Path, dst: pathlib.Path) -> bool: """Move file and verify by hash""" src_hash = calculate_file_hash(src) src.replace(dst) # atomic move (overwrite if exists) dst_hash = calculate_file_hash(dst) return src_hash == dst_hash def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime) -> bool: """Process a pair of PDF (in Images/) and XML files""" pdf_created_at = datetime.fromtimestamp(pdf_file.stat().st_ctime) if pdf_created_at > time_threshold: logger.debug(f"Skipping {pdf_file.name} (created {pdf_created_at:%Y-%m-%d %H:%M})") return False # Construct target paths rel_path = pdf_file.relative_to(pdf_file.parents[1]) # Images/subdir/file.pdf -> subdir/file.pdf target_pdf_path = target_folder / "Images" / rel_path target_xml_path = target_folder / xml_file.name # Create target dirs if needed target_pdf_path.parent.mkdir(parents=True, exist_ok=True) # Move files and verify pdf_moved = move_file(pdf_file, target_pdf_path) xml_moved = move_file(xml_file, target_xml_path) if pdf_moved and xml_moved: logger.info(f"Moved {pdf_file.name} and {xml_file.name} to {target_folder}") return True else: logger.error(f"Hash mismatch or move failed for {pdf_file.name} and/or {xml_file.name}") # Restore original files (optional, but nice for debugging) # target_pdf_path.replace(pdf_file) # target_xml_path.replace(xml_file) return False def main(folder_a: pathlib.Path, folder_b: pathlib.Path, days: int): logger.info(f"Moving files older than {days} days from {folder_a} to {folder_b}") time_threshold = datetime.now() - timedelta(days=days) # Find all PDF files in Images/ subfolders pdf_files = list((folder_a / "Images").rglob("*.pdf")) logger.info(f"Found {len(pdf_files)} PDF files") successes = 0 for pdf_file in tqdm(pdf_files, desc="Processing file pairs"): xml_file = folder_a / f"{pdf_file.stem}.xml" if not xml_file.exists(): logger.warning(f"No matching XML file for {pdf_file.name}") continue if process_file_pair(pdf_file, xml_file, folder_b, time_threshold): successes += 1 logger.info(f"Moved {successes} file pairs successfully") if __name__ == '__main__': folder_a = pathlib.Path(r"\\network\folder\source") # or simply r"C:\local\folder" folder_b = pathlib.Path(r"C:\target\folder") days = 14 # 2 weeks logger.add("file_mover.log", rotation="1 week", level="DEBUG") # log to file logger.info("Starting file mover script") main(folder_a, folder_b, days)Let's go through the key aspects:
calculate_file_hash: computes the SHA-256 hash of a file in chunks (efficient for large files).move_file: moves a file atomically (usingreplace) and verifies the move by comparing hashes. ReturnsTrueif successful.process_file_pair: the core logic. It:- Checks the PDF file's creation time (Windows:
st_ctime). - If older than the threshold, constructs target paths for PDF and XML files.
- Creates target dirs if needed.
- Moves both files using
move_file, verifying each move. - Logs success or failure.
- Checks the PDF file's creation time (Windows:
main: orchestrates the process:- Finds all PDF files in
Images/subfolders usingrglob. - Iterates through PDF files, finds matching XML files, and calls
process_file_pair. - Logs overall statistics.
- Finds all PDF files in
Notes:
pathlibmakes working with paths delightful. Note the use ofrglob,relative_to, andparentsto manipulate paths.loguruis a fantastic logging library. Here, we log to a rotating file (file_mover.log) and the console.tqdmprovides a nice progress bar for the file iteration.- We use
datetimeandtimedeltato calculate the time threshold (2 weeks ago). - The script assumes your XML files are directly under
folder_a, while PDFs are infolder_a/Images/. - Network folders: Make sure the Python process has the necessary permissions to read from
folder_aand write tofolder_b. If you encounter issues, try mapping the network folder to a drive letter (e.g.,Z:\) and use that path in the script. - Testing: Before running the script on your actual data, test it on a small subset of files to ensure everything works as expected.
Run the script, and you should see logging output on the console and in the
file_mover.logfile. Happy file moving! - Python 3.8+ (for
- User
Why not copy the file first, compare hashes and only delete the file if the hashes match? and good idea to use tqdm
- Assistant
Copy-then-verify-then-delete is a much safer and more robust approach than move-then-verify. Here's why:
- Atomicity is not enough: While
pathlib.Path.replace()(oros.replace()) is atomic, meaning it either completes fully or not at all (no partial writes), it doesn't guarantee data integrity. A successfulreplace()only means the file system metadata was updated correctly; it doesn't verify the data was written correctly. - Storage errors can happen: Between the move (metadata update) and the actual data block write, errors can occur (e.g., disk I/O issues, file system corruption, power loss). If that happens, you might end up with:
- A file seemingly moved (metadata updated).
- But the data blocks not fully written or corrupted.
- Your verification step (hash check) fails, but now the original file is gone (because you moved it).
- Data loss.
- Copy, verify, delete: The safe three-step:
- Copy the file (e.g.,
shutil.copy2()orpathlib.Path.write_bytes()after reading the source). This writes the full data to the new location. - Verify (hash check) that the copied data matches the original data. If hashes match, you know the copy is byte-for-byte identical.
- Only then delete the original file. If verification fails, you still have the original, and you can retry or log the error.
- Copy the file (e.g.,
This approach is especially crucial when:
- Dealing with network storage (SMB/CIFS/NFS): network hiccups can corrupt data during transfer.
- Using slow or unreliable storage (e.g., old disks, USB sticks).
- Handling large files where a partial write is disastrous.
Now, let's update the script to follow this Copy → Verify → Delete (if OK) workflow. We'll also keep
tqdmfor that nice progress bar.The Updated Script:
import pathlib from datetime import datetime, timedelta import hashlib import shutil from loguru import logger from tqdm import tqdm def calculate_file_hash(file_path: pathlib.Path) -> str: """Calculate SHA-256 hash of a file""" hash = hashlib.sha256() with file_path.open('rb') as f: while chunk := f.read(8192): hash.update(chunk) return hash.hexdigest() def copy_and_verify(src: pathlib.Path, dst: pathlib.Path) -> bool: """Copy file and verify by hash""" src_hash = calculate_file_hash(src) shutil.copy2(src, dst) # preserves metadata (ctime, mtime, etc.) dst_hash = calculate_file_hash(dst) return src_hash == dst_hash def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime) -> bool: """Process a pair of PDF (in Images/) and XML files""" pdf_created_at = datetime.fromtimestamp(pdf_file.stat().st_ctime) if pdf_created_at > time_threshold: logger.debug(f"Skipping {pdf_file.name} (created {pdf_created_at:%Y-%m-%d %H:%M})") return False # Construct target paths rel_path = pdf_file.relative_to(pdf_file.parents[1]) # Images/subdir/file.pdf -> subdir/file.pdf target_pdf_path = target_folder / "Images" / rel_path target_xml_path = target_folder / xml_file.name # Create target dirs if needed target_pdf_path.parent.mkdir(parents=True, exist_ok=True) # Copy and verify both files pdf_ok = copy_and_verify(pdf_file, target_pdf_path) xml_ok = copy_and_verify(xml_file, target_xml_path) if pdf_ok and xml_ok: logger.info(f"Copied {pdf_file.name} and {xml_file.name} to {target_folder} successfully") # Now it's safe to delete originals pdf_file.unlink() xml_file.unlink() logger.debug(f"Deleted originals: {pdf_file.name} and {xml_file.name}") return True else: logger.error(f"Hash mismatch or copy failed for {pdf_file.name} and/or {xml_file.name}. Originals kept.") # Optional: remove incomplete targets to avoid confusion # if target_pdf_path.exists(): target_pdf_path.unlink() # if target_xml_path.exists(): target_xml_path.unlink() return False def main(folder_a: pathlib.Path, folder_b: pathlib.Path, days: int): logger.info(f"Moving files older than {days} days from {folder_a} to {folder_b}") time_threshold = datetime.now() - timedelta(days=days) pdf_files = list((folder_a / "Images").rglob("*.pdf")) logger.info(f"Found {len(pdf_files)} PDF files") successes = 0 for pdf_file in tqdm(pdf_files, desc="Processing file pairs"): xml_file = folder_a / f"{pdf_file.stem}.xml" if not xml_file.exists(): logger.warning(f"No matching XML file for {pdf_file.name}") continue if process_file_pair(pdf_file, xml_file, folder_b, time_threshold): successes += 1 logger.info(f"Copied and deleted {successes} file pairs successfully") if __name__ == '__main__': folder_a = pathlib.Path(r"\\network\folder\source") folder_b = pathlib.Path(r"C:\target\folder") days = 14 logger.add("file_mover.log", rotation="1 week", level="DEBUG") logger.info("Starting safe file mover script (copy-verify-delete)") main(folder_a, folder_b, days)Changes:
copy_and_verify: replacesmove_file. It usesshutil.copy2()(which preserves metadata likectime) to copy the file, then checks hashes.- In
process_file_pair:- We copy both PDF and XML files first.
- If both copies verify (hashes match), then we delete the originals.
- If verification fails, we log the error and keep the originals. Optionally, you can remove the incomplete target files.
Why
shutil.copy2()?shutil.copy()copies data, but not necessarily all metadata (e.g., creation time might not be preserved on Windows).shutil.copy2()tries to preserve all available metadata (mode, timestamps, etc.). On Windows, this includesst_ctime(creation time), which is crucial if you rely on that for your logic.
Now, even if something goes wrong during the copy (disk full, network glitch, cosmic rays 🔥), your original files remain intact until the successful verification step. Much safer!
Run this updated script, and you'll sleep better knowing your data is handled with care 😊. Thanks again for the suggestion!
- Atomicity is not enough: While
- User
add a flag which toggles if the files get deleted or not, so it is easier to try out. also add a cmd interface using argparse
- Assistant
Excellent suggestions! Let's enhance the script with:
- A
--dry-run(or--no-delete) flag: This will copy and verify, but not delete the original files. Perfect for testing without risking data loss. - A
--delete(or--confirm-delete) flag: The opposite of dry-run. When set, it will delete the originals after successful copy+verify. This makes it explicit when you're ready to actually move files. - A Command-Line Interface (CLI) using
argparse: This lets you run the script from the terminal/command prompt with intuitive arguments, instead of editing the script each time.
Here's the updated code:
import pathlib from datetime import datetime, timedelta import hashlib import shutil from loguru import logger from tqdm import tqdm import argparse def calculate_file_hash(file_path: pathlib.Path) -> str: """Calculate SHA-256 hash of a file""" hash = hashlib.sha256() with file_path.open('rb') as f: while chunk := f.read(8192): hash.update(chunk) return hash.hexdigest() def copy_and_verify(src: pathlib.Path, dst: pathlib.Path) -> bool: """Copy file and verify by hash""" src_hash = calculate_file_hash(src) shutil.copy2(src, dst) # preserves metadata dst_hash = calculate_file_hash(dst) return src_hash == dst_hash def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime, delete_after: bool) -> bool: """Process a pair of PDF (in Images/) and XML files""" pdf_created_at = datetime.fromtimestamp(pdf_file.stat().st_ctime) if pdf_created_at > time_threshold: logger.debug(f"Skipping {pdf_file.name} (created {pdf_created_at:%Y-%m-%d %H:%M})") return False rel_path = pdf_file.relative_to(pdf_file.parents[1]) # Images/subdir/file.pdf -> subdir/file.pdf target_pdf_path = target_folder / "Images" / rel_path target_xml_path = target_folder / xml_file.name target_pdf_path.parent.mkdir(parents=True, exist_ok=True) pdf_ok = copy_and_verify(pdf_file, target_pdf_path) xml_ok = copy_and_verify(xml_file, target_xml_path) if pdf_ok and xml_ok: logger.info(f"Copied {pdf_file.name} and {xml_file.name} to {target_folder} successfully") if delete_after: pdf_file.unlink() xml_file.unlink() logger.info(f"Deleted originals: {pdf_file.name} and {xml_file.name}") else: logger.info(f"Dry-run: Originals NOT deleted (use --delete to confirm deletion)") return True else: logger.error(f"Hash mismatch or copy failed for {pdf_file.name} and/or {xml_file.name}. No deletion attempted.") # Optional: cleanup failed targets # if target_pdf_path.exists(): target_pdf_path.unlink() # if target_xml_path.exists(): target_xml_path.unlink() return False def main(args): src_folder = pathlib.Path(args.source) dst_folder = pathlib.Path(args.target) days = args.days delete_files = args.delete logger.info(f"Script mode: {'DELETE ORIGINAL FILES' if delete_files else 'DRY RUN (no deletion)'}") logger.info(f"Moving files older than {days} days from {src_folder} to {dst_folder}") time_threshold = datetime.now() - timedelta(days=days) pdf_files = list((src_folder / "Images").rglob("*.pdf")) logger.info(f"Found {len(pdf_files)} PDF files") successes = 0 for pdf_file in tqdm(pdf_files, desc="Processing file pairs"): xml_file = src_folder / f"{pdf_file.stem}.xml" if not xml_file.exists(): logger.warning(f"No matching XML file for {pdf_file.name}") continue if process_file_pair(pdf_file, xml_file, dst_folder, time_threshold, delete_files): successes += 1 logger.info(f"Copied {'and deleted' if delete_files else 'but did NOT delete'} {successes} file pairs successfully") if __name__ == '__main__': parser = argparse.ArgumentParser(description="Safe File Mover: Copy PDF+XML pairs older than X days, verify by hash, optionally delete originals.") parser.add_argument("-s", "--source", required=True, type=pathlib.Path, help="Source folder (containing Images/ subfolder)") parser.add_argument("-t", "--target", required=True, type=pathlib.Path, help="Target folder (where files will be copied)") parser.add_argument("-d", "--days", type=int, default=14, help="Age threshold in days (default: 14)") group = parser.add_mutually_exclusive_group() group.add_argument("--delete", action="store_true", help="Confirm deletion of original files after successful copy+verify (DANGER: irreversible!)") group.add_argument("--dry-run", action="store_false", dest="delete", help="Test mode: copy+verify, but DO NOT DELETE originals (default)") parser.set_defaults(delete=False) # Default to dry-run args = parser.parse_args() logger.add("file_mover.log", rotation="1 week", level="DEBUG") logger.info("Starting file mover script") logger.debug(f"Args: {args}") if not args.source.exists() or not (args.source / "Images").exists(): logger.error(f"Source folder {args.source} or {args.source / 'Images'} does not exist!") exit(1) if not args.target.exists(): logger.warning(f"Target folder {args.target} does not exist. Will create it.") args.target.mkdir(parents=True, exist_ok=True) main(args)What's new:
argparsesets up a CLI with:--source(-s): required, the folder containingImages/.--target(-t): required, where files will be copied.--days(-d): optional (default=14), the age threshold.- Mutually exclusive group:
--delete: Enable deletion of originals after copy+verify. Use with caution!--dry-run: Disable deletion (default). Just copy+verify, don't touch originals.
process_file_pairnow takes an extradelete_after: boolargument. IfTrue, it deletes originals on success.- In
main():- We parse
args.deleteto decide whether to delete files. - Log the script mode clearly: "DRY RUN" or "DELETE ORIGINAL FILES".
- We parse
- Validation:
- Check if
source/andsource/Images/exist. - If
target/doesn't exist, create it (with a warning).
- Check if
Example usage:
# DRY RUN (default): Copy but don't delete python file_mover.py -s \\network\old_files -t D:\archive --days 30 # DELETE ORIGINALS (be careful!) python file_mover.py -s \\network\old_files -t D:\archive --days 30 --delete # Help screen python file_mover.py -hOutput of
python file_mover.py -h:usage: file_mover.py [-h] -s SOURCE -t TARGET [-d DAYS] [--delete | --dry-run] Safe File Mover: Copy PDF+XML pairs older than X days, verify by hash, optionally delete originals. optional arguments: -h, --help show this help message and exit -s SOURCE, --source SOURCE Source folder (containing Images/ subfolder) -t TARGET, --target TARGET Target folder (where files will be copied) -d DAYS, --days DAYS Age threshold in days (default: 14) --delete Confirm deletion of original files after successful copy+verify (DANGER: irreversible!) --dry-run Test mode: copy+verify, but DO NOT DELETE originals (default)Now you can test freely with
--dry-run(the default), verify everything works, and then switch to--deletewhen you're confident.Happy (safe) file moving! 🚀
- A
- User
Check if dest file exists and has same hash, if so skip copy
- Assistant
An excellent optimization! Let's add a "destination exists and matches" check before copying. This way:
- If the destination file already exists (e.g., from a previous run or manual copy).
- And its hash matches the source file's hash.
- We skip the copy entirely (no need to re-transfer data).
- We still delete the source (since the data is safely duplicated).
This tweak saves:
- Bandwidth (if copying over network).
- Disk I/O (if local, but still slower than a simple
stat()check). - Time (especially for large files).
Here's the updated
copy_and_verifyfunction:def copy_and_verify(src: pathlib.Path, dst: pathlib.Path) -> bool: """Copy file if needed (dst missing or hash mismatch) and verify""" src_hash = calculate_file_hash(src) if dst.exists(): dst_hash = calculate_file_hash(dst) if src_hash == dst_hash: logger.debug(f"Skipping copy of {src.name}: Destination exists and matches") return True # Destination already has the correct file else: logger.warning(f"Destination {dst.name} exists BUT HASH MISSMATCH. Replacing...") dst.unlink() # Remove outdated/incorrect file # Either dst didn't exist or was wrong; copy now shutil.copy2(src, dst) dst_hash_after_copy = calculate_file_hash(dst) if src_hash == dst_hash_after_copy: logger.debug(f"Copied {src.name} to {dst.parent} and verified OK") return True else: logger.error(f"Hash mismatch after copying {src.name} to {dst}! Data corruption?") # Optional: dst.unlink() # Cleanup failed copy? return FalseHow it works:
- Calculate the
src_hash(once). - Check if
dstexists:- If yes, calculate
dst_hash. - If
src_hash == dst_hash, log a debug message and returnTrue(no copy needed; data is already there and correct). - If hashes mismatch, log a warning, delete
dst(it's outdated or corrupt), and proceed to copy.
- If yes, calculate
- If
dstdoesn't exist (or was deleted due to hash mismatch), copysrc→dstusingshutil.copy2(). - Re-calculate
dst_hash(after copy) and verify it matchessrc_hash.- If match, all good! Return
True. - If mismatch, log an error (data corruption during copy?). Return
False.
- If match, all good! Return
The rest of the script remains the same. Now,
process_file_pairstill callscopy_and_verifyfor both PDF and XML files, but the new logic reduces unnecessary copies:def process_file_pair(pdf_file: pathlib.Path, xml_file: pathlib.Path, target_folder: pathlib.Path, time_threshold: datetime) -> bool: # ... (age check and path construction as before) pdf_ok = copy_and_verify(pdf_file, target_pdf_path) xml_ok = copy_and_verify(xml_file, target_xml_path) if pdf_ok and xml_ok: logger.info(f"Files {pdf_file.name} and {xml_file.name} are safely duplicated in {target_folder}") pdf_file.unlink() xml_file.unlink() logger.debug(f"Deleted originals: {pdf_file.name} and {xml_file.name}") return True else: logger.error(f"Copy/verify failed for {pdf_file.name} and/or {xml_file.name}. Originals kept.") # Optional: cleanup target_pdf_path, target_xml_path if exists return FalseBenefits:
- Faster reruns: If you re-run the script (e.g., with different
daysthreshold), files that were already copied correctly aren't recopied. - Network friendlier: No redundant GBs transferred over the wire.
- Disk sparing: Less write wear (though modern SSDs handle this well, it's still a nice optimization).
Edge case: If someone manually modifies the destination file (after our script copied it), the hashes will mismatch next run, and we'll re-copy the original. This keeps your archive consistent with the sources.
Run this refined version, and you'll enjoy:
- Fewer logs about successful copies (since skips are logged at
DEBUG). - Same data integrity guarantees.
- Better performance on subsequent executions.
Happy (smarter) file moving 🚀!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.