Modern python with pathlib etc. I work on windows. I want to move files older than 2…
- User
Modern python with pathlib etc. I work on windows. I want to move files older than 2 Weeks (by created by) from one folder (potentially a network folder) to a different folder. I want to delete the old file, but only after checking that the move/copy was successfull (by comparing hashs for example). Inputs are folder a, b and the amount of time passed to copy the files. There are actually two files, and I only want to copy the pairs. one is in folder Images, and has .pdf extension, one is in the main folder and has the same name but .xml extension. Keep this Images folder structure. Overwrite existing files. Log using loguru
Assistant
To accomplish the task you described, we'll use Python with
pathlibfor handling paths,logurufor logging,hashlibfor calculating file hashes, andshutilfor moving files. The script will be designed to work on Windows and will be able to handle network folders.First, ensure you have the necessary packages installed. You can install them using pip if you haven't already:
pip install loguruHere's a Python script that meets your requirements:
import pathlib import loguru from datetime import datetime, timedelta import hashlib import shutil # Initialize logger logger = loguru.logger def calculate_file_hash(file_path): """Calculates the SHA-256 hash of a file.""" sha256_hash = hashlib.sha256() with open(file_path, "rb") as f: # Read and update hash in chunks of 4K for byte_block in iter(lambda: f.read(4096), b""): sha256_hash.update(byte_block) return sha256_hash.hexdigest() def move_file_if_valid(source, destination): """Moves a file and checks if the move was successful by comparing hashes.""" try: source_hash = calculate_file_hash(source) shutil.move(str(source), str(destination)) destination_hash = calculate_file_hash(destination) if source_hash == destination_hash: logger.info(f"Successfully moved {source} to {destination}") return True else: logger.error(f"Hash mismatch after moving {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}") # Attempt to recover or handle the error return False except Exception as e: logger.error(f"Error moving {source} to {destination}: {e}") return False def main(folder_a, folder_b, days_threshold): folder_a = pathlib.Path(folder_a) folder_b = pathlib.Path(folder_b) images_folder = folder_a / 'Images' cutoff_date = datetime.now() - timedelta(days=days_threshold) for xml_file in folder_a.glob('*.xml'): pdf_file = images_folder / (xml_file.stem + '.pdf') if pdf_file.exists(): xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime) if xml_created_time < cutoff_date: relative_path = pdf_file.relative_to(images_folder) destination_pdf = folder_b / 'Images' / relative_path destination_xml = folder_b / xml_file.name # Ensure destination directories exist destination_pdf.parent.mkdir(parents=True, exist_ok=True) destination_xml.parent.mkdir(parents=True, exist_ok=True) # Move files move_file_if_valid(pdf_file, destination_pdf) move_file_if_valid(xml_file, destination_xml) else: logger.warning(f"No corresponding PDF found for {xml_file}") if __name__ == "__main__": folder_a = r"\\path\to\source\folder" # Update this path folder_b = r"\\path\to\destination\folder" # Update this path days_threshold = 14 # 2 weeks main(folder_a, folder_b, days_threshold)Key Points:
- The script uses
pathlibfor path manipulations, making it easy to work with Windows paths and network locations. - It checks for the existence of corresponding
.pdfand.xmlfiles and only moves them if both exist. - The
move_file_if_validfunction moves a file and verifies the move by comparing the source and destination file hashes. - The script logs its actions and any errors encountered using
loguru. - It preserves the structure of the 'Images' folder when moving files.
- You can adjust
folder_a,folder_b, anddays_thresholdin theif __name__ == "__main__":block to suit your needs.
Before Running:
- Replace
\\path\to\source\folderand\\path\to\destination\folderwith your actual source and destination folder paths. - Ensure the Python script has the necessary permissions to read from the source folder and write to the destination folder.
This script should be a good starting point for your task. You might need to tweak it based on your specific requirements or environment.
- The script uses
- User
Why not copy the file first, compare hashes and only delete the file if the hashes match? and good idea to use tqdm
Assistant
Copying the file first and then comparing hashes is a much safer approach than moving the file and then checking the hash. If the hash comparison fails after moving, you've already lost the original file. By copying first, you ensure that the original data is preserved until you've verified the copy.
Here's an updated version of the script that incorporates this approach and also adds
tqdmfor progress indication:import pathlib import loguru from datetime import datetime, timedelta import hashlib import shutil from tqdm import tqdm # Initialize logger logger = loguru.logger def calculate_file_hash(file_path): """Calculates the SHA-256 hash of a file.""" sha256_hash = hashlib.sha256() with open(file_path, "rb") as f: # Read and update hash in chunks of 4K for byte_block in iter(lambda: f.read(4096), b""): sha256_hash.update(byte_block) return sha256_hash.hexdigest() def copy_and_move_file_if_valid(source, destination): """Copies a file, checks if the copy was successful by comparing hashes, and then removes the original if successful.""" try: source_hash = calculate_file_hash(source) shutil.copy2(str(source), str(destination)) # Copy with metadata destination_hash = calculate_file_hash(destination) if source_hash == destination_hash: logger.info(f"Successfully copied {source} to {destination}") source.unlink() # Remove the original file logger.info(f"Removed original file {source}") else: logger.error(f"Hash mismatch after copying {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}") destination.unlink() # Remove the failed copy except Exception as e: logger.error(f"Error copying {source} to {destination}: {e}") def main(folder_a, folder_b, days_threshold): folder_a = pathlib.Path(folder_a) folder_b = pathlib.Path(folder_b) images_folder = folder_a / 'Images' cutoff_date = datetime.now() - timedelta(days=days_threshold) xml_files = list(folder_a.glob('*.xml')) logger.info(f"Found {len(xml_files)} XML files in {folder_a}") for xml_file in tqdm(xml_files, desc="Processing XML files"): pdf_file = images_folder / (xml_file.stem + '.pdf') if pdf_file.exists(): xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime) if xml_created_time < cutoff_date: relative_path = pdf_file.relative_to(images_folder) destination_pdf = folder_b / 'Images' / relative_path destination_xml = folder_b / xml_file.name # Ensure destination directories exist destination_pdf.parent.mkdir(parents=True, exist_ok=True) destination_xml.parent.mkdir(parents=True, exist_ok=True) # Copy and potentially remove original files copy_and_move_file_if_valid(pdf_file, destination_pdf) copy_and_move_file_if_valid(xml_file, destination_xml) else: logger.warning(f"No corresponding PDF found for {xml_file}") if __name__ == "__main__": folder_a = r"\\path\to\source\folder" # Update this path folder_b = r"\\path\to\destination\folder" # Update this path days_threshold = 14 # 2 weeks main(folder_a, folder_b, days_threshold)Changes:
- The
copy_and_move_file_if_validfunction now copies the file first, compares hashes, and only removes the original file if the hashes match. tqdmis used to display a progress bar when processing XML files.- The script logs the number of XML files found at the start.
This updated approach ensures data integrity by verifying the copy before removing the original.
- The
- User
add a flag which toggles if the files get deleted or not, so it is easier to try out. also add a cmd interface using argparse
Assistant
Here's the updated script with a flag to toggle file deletion and a command-line interface using
argparse:import pathlib import loguru from datetime import datetime, timedelta import hashlib import shutil from tqdm import tqdm import argparse # Initialize logger logger = loguru.logger def calculate_file_hash(file_path): """Calculates the SHA-256 hash of a file.""" sha256_hash = hashlib.sha256() with open(file_path, "rb") as f: # Read and update hash in chunks of 4K for byte_block in iter(lambda: f.read(4096), b""): sha256_hash.update(byte_block) return sha256_hash.hexdigest() def copy_and_move_file_if_valid(source, destination, delete_original): """Copies a file, checks if the copy was successful by comparing hashes, and then removes the original if successful and delete_original is True.""" try: source_hash = calculate_file_hash(source) shutil.copy2(str(source), str(destination)) # Copy with metadata destination_hash = calculate_file_hash(destination) if source_hash == destination_hash: logger.info(f"Successfully copied {source} to {destination}") if delete_original: source.unlink() # Remove the original file logger.info(f"Removed original file {source}") else: logger.info(f"Skipping removal of original file {source} due to --dry-run") else: logger.error(f"Hash mismatch after copying {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}") destination.unlink() # Remove the failed copy except Exception as e: logger.error(f"Error copying {source} to {destination}: {e}") def main(folder_a, folder_b, days_threshold, delete_original): folder_a = pathlib.Path(folder_a) folder_b = pathlib.Path(folder_b) images_folder = folder_a / 'Images' cutoff_date = datetime.now() - timedelta(days=days_threshold) xml_files = list(folder_a.glob('*.xml')) logger.info(f"Found {len(xml_files)} XML files in {folder_a}") for xml_file in tqdm(xml_files, desc="Processing XML files"): pdf_file = images_folder / (xml_file.stem + '.pdf') if pdf_file.exists(): xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime) if xml_created_time < cutoff_date: relative_path = pdf_file.relative_to(images_folder) destination_pdf = folder_b / 'Images' / relative_path destination_xml = folder_b / xml_file.name # Ensure destination directories exist destination_pdf.parent.mkdir(parents=True, exist_ok=True) destination_xml.parent.mkdir(parents=True, exist_ok=True) # Copy and potentially remove original files copy_and_move_file_if_valid(pdf_file, destination_pdf, delete_original) copy_and_move_file_if_valid(xml_file, destination_xml, delete_original) else: logger.warning(f"No corresponding PDF found for {xml_file}") if __name__ == "__main__": parser = argparse.ArgumentParser(description="Copy and optionally remove old XML and PDF files") parser.add_argument("-a", "--source", required=True, help="Source folder path") parser.add_argument("-b", "--destination", required=True, help="Destination folder path") parser.add_argument("-d", "--days", type=int, required=True, help="Number of days to consider a file old") parser.add_argument("--dry-run", action="store_false", dest="delete_original", help="Copy files but do not remove originals (default)") parser.add_argument("--delete", action="store_true", dest="delete_original", help="Copy files and remove originals") parser.add_argument("-v", "--verbose", action="store_true", help="Increase verbosity (not implemented)") args = parser.parse_args() if args.delete_original: logger.info("Files will be deleted after successful copy") else: logger.info("Files will not be deleted after copy (dry run)") main(args.source, args.destination, args.days, args.delete_original)Changes:
- Added a
--dry-runflag (default) and a--deleteflag to toggle the deletion of original files after copying. - Implemented a command-line interface using
argparseto accept source folder, destination folder, number of days, and the delete flag as arguments. - The script now logs whether it will delete files or not based on the
--deleteor--dry-runflag.
You can run the script from the command line like this:
python script_name.py -a /path/to/source -b /path/to/destination -d 14 --deleteOr for a dry run:
python script_name.py -a /path/to/source -b /path/to/destination -d 14- Added a
- User
Check if dest file exists and has same hash, if so skip copy
Assistant
Here's an updated version of the script that checks if the destination file exists and has the same hash before copying:
import pathlib import loguru from datetime import datetime, timedelta import hashlib import shutil from tqdm import tqdm # Initialize logger logger = loguru.logger def calculate_file_hash(file_path): """Calculates the SHA-256 hash of a file.""" sha256_hash = hashlib.sha256() with open(file_path, "rb") as f: # Read and update hash in chunks of 4K for byte_block in iter(lambda: f.read(4096), b""): sha256_hash.update(byte_block) return sha256_hash.hexdigest() def copy_and_move_file_if_valid(source, destination, delete_original): """Copies a file, checks if the copy was successful by comparing hashes, and then removes the original if successful and delete_original is True.""" try: if destination.exists(): source_hash = calculate_file_hash(source) destination_hash = calculate_file_hash(destination) if source_hash == destination_hash: logger.info(f"Skipping copy of {source} to {destination} as it already exists with the same hash") if delete_original: source.unlink() # Remove the original file logger.info(f"Removed original file {source}") return else: logger.info(f"Destination file {destination} exists but has a different hash. Overwriting...") source_hash = calculate_file_hash(source) shutil.copy2(str(source), str(destination)) # Copy with metadata destination_hash = calculate_file_hash(destination) if source_hash == destination_hash: logger.info(f"Successfully copied {source} to {destination}") if delete_original: source.unlink() # Remove the original file logger.info(f"Removed original file {source}") else: logger.error(f"Hash mismatch after copying {source} to {destination}. Source hash: {source_hash}, Destination hash: {destination_hash}") destination.unlink() # Remove the failed copy except Exception as e: logger.error(f"Error copying {source} to {destination}: {e}") def main(folder_a, folder_b, days_threshold, delete_original): folder_a = pathlib.Path(folder_a) folder_b = pathlib.Path(folder_b) images_folder = folder_a / 'Images' cutoff_date = datetime.now() - timedelta(days=days_threshold) xml_files = list(folder_a.glob('*.xml')) logger.info(f"Found {len(xml_files)} XML files in {folder_a}") for xml_file in tqdm(xml_files, desc="Processing XML files"): pdf_file = images_folder / (xml_file.stem + '.pdf') if pdf_file.exists(): xml_created_time = datetime.fromtimestamp(xml_file.stat().st_ctime) if xml_created_time < cutoff_date: relative_path = pdf_file.relative_to(images_folder) destination_pdf = folder_b / 'Images' / relative_path destination_xml = folder_b / xml_file.name # Ensure destination directories exist destination_pdf.parent.mkdir(parents=True, exist_ok=True) destination_xml.parent.mkdir(parents=True, exist_ok=True) # Copy and potentially remove original files copy_and_move_file_if_valid(pdf_file, destination_pdf, delete_original) copy_and_move_file_if_valid(xml_file, destination_xml, delete_original) else: logger.warning(f"No corresponding PDF found for {xml_file}") if __name__ == "__main__": folder_a = r"\\path\to\source\folder" # Update this path folder_b = r"\\path\to\destination\folder" # Update this path days_threshold = 14 # 2 weeks delete_original = True # Set to False for a dry run main(folder_a, folder_b, days_threshold, delete_original)Changes:
- The
copy_and_move_file_if_validfunction now checks if the destination file exists and has the same hash as the source file. If so, it skips the copy operation. - The
delete_originalparameter allows you to control whether the original files are deleted after a successful copy. Set it toFalsefor a dry run.
- The
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.