Skip to content
TheChatArchiveThe Chat Archive
Grok 4.1 Fast· xaiHugging Face Frontier Reasoning Traces

Write a Python function to find duplicate files in a directory tree.

  1. User

    Write a Python function to find duplicate files in a directory tree.

  2. Assistant
    import os
    import hashlib
    from collections import defaultdict
    from typing import List
    
    def find_duplicate_files(root_dir: str) -> List[List[str]]:
        """
        Find duplicate files in a directory tree by comparing their SHA-256 hashes.
        
        Args:
            root_dir (str): The root directory to scan.
        
        Returns:
            List[List[str]]: A list of lists, where each inner list contains paths
            to files that are duplicates (same content). Only groups with 2+ files
            are included. Paths in each group are sorted alphabetically.
        
        Notes:
            - Skips files that cannot be read (e.g., permission denied).
            - Does not follow symbolic links.
            - Efficiently hashes files in chunks to handle large files.
        """
        if not os.path.exists(root_dir):
            return []
        
        file_hashes = defaultdict(list)
        
        def get_file_hash(file_path: str) -> str:
            hash_obj = hashlib.sha256()
            try:
                with open(file_path, 'rb') as f:
                    for chunk in iter(lambda: f.read(4096), b''):
                        hash_obj.update(chunk)
                return hash_obj.hexdigest()
            except (IOError, OSError):
                return None
        
        for dirpath, _, filenames in os.walk(root_dir, followlinks=False):
            for filename in filenames:
                file_path = os.path.join(dirpath, filename)
                file_hash = get_file_hash(file_path)
                if file_hash:
                    file_hashes[file_hash].append(file_path)
        
        # Filter groups with duplicates and sort paths in each group
        duplicates = [
            sorted(paths) for paths in file_hashes.values()
            if len(paths) > 1
        ]
        
        return duplicates
    

    Example Usage:

    # Find duplicates in current directory
    dups = find_duplicate_files('.')
    for group in dups:
        print("Duplicates:")
        for path in group:
            print(f"  - {path}")
    

    This function is efficient, handles errors gracefully, and returns results in a clean, sorted format.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.