Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-5-sonnet-20241022· AnthropicHugging Face LMArena Human Preference 140K

I have a PDF file that is a concatenation of several scanned documents. How to implement…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash’s answer.
  1. User

    I have a PDF file that is a concatenation of several scanned documents. How to implement a solution in Python that enables me to recover the original documents ?

  2. Claude Assistant

    Here's a solution to split a PDF containing scanned documents into separate files based on visual analysis and content detection:

    import cv2
    import numpy as np
    import fitz  # PyMuPDF
    from PIL import Image
    import io
    import os
    
    def convert_page_to_image(page):
        """Convert PDF page to numpy array image"""
        pix = page.get_pixmap()
        img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
        return np.array(img)
    
    def is_blank_page(image, threshold=0.95):
        """Detect if a page is mostly blank"""
        gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
        white_pixels = np.sum(gray > 250)
        total_pixels = gray.size
        return (white_pixels / total_pixels) > threshold
    
    def detect_document_boundary(image):
        """Detect if page likely represents start of new document"""
        # Convert to grayscale
        gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
        
        # Apply threshold
        _, thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
        
        # Look for common indicators of new document:
        # 1. Large text at top of page
        # 2. Significant white space followed by content
        # 3. Distinct header patterns
        
        # Simple heuristic: Check top portion of page for significant content
        top_portion = thresh[:int(thresh.shape[0] * 0.2), :]
        content_density = np.sum(top_portion < 128) / top_portion.size
        
        return content_density > 0.1  # Adjust threshold as needed
    
    def split_pdf(input_path, output_dir):
        """Split PDF into separate documents"""
        # Create output directory if it doesn't exist
        os.makedirs(output_dir, exist_ok=True)
        
        # Open PDF
        doc = fitz.open(input_path)
        
        current_doc_pages = []
        doc_counter = 1
        
        def save_current_doc():
            nonlocal doc_counter
            if current_doc_pages:
                new_doc = fitz.open()
                for page_num in current_doc_pages:
                    new_doc.insert_pdf(doc, from_page=page_num, to_page=page_num)
                output_path = os.path.join(output_dir, f'document_{doc_counter:03d}.pdf')
                new_doc.save(output_path)
                new_doc.close()
                doc_counter += 1
                current_doc_pages.clear()
        
        for page_num in range(len(doc)):
            page = doc[page_num]
            image = convert_page_to_image(page)
            
            # Skip blank pages
            if is_blank_page(image):
                continue
            
            # If this looks like the start of a new document and we have existing pages,
            # save the current document
            if detect_document_boundary(image) and current_doc_pages:
                save_current_doc()
            
            current_doc_pages.append(page_num)
        
        # Save the last document
        save_current_doc()
        
        doc.close()
    
    def main():
        input_pdf = "scanned_documents.pdf"
        output_dir = "split_documents"
        
        print("Starting PDF splitting process...")
        split_pdf(input_pdf, output_dir)
        print(f"Documents have been split and saved to {output_dir}/")
    
    if __name__ == "__main__":
        main()
    

    This solution includes several key features:

    1. Page Analysis:

      • Converts PDF pages to images for analysis
      • Detects blank pages
      • Identifies potential document boundaries
    2. Document Boundary Detection:

      • Looks for typical indicators of new documents
      • Analyzes content density in the top portion of pages
      • Can be enhanced with more sophisticated detection methods
    3. Memory Efficiency:

      • Processes one page at a time
      • Uses PyMuPDF (fitz) for efficient PDF handling

    To make this more robust, you could add these enhancements:

    def enhance_boundary_detection(image):
        """Enhanced document boundary detection"""
        # Convert to grayscale
        gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
        
        # Apply OCR to detect headers or title patterns
        try:
            import pytesseract
            text = pytesseract.image_to_string(gray)
            
            # Look for keywords indicating new document
            keywords = ['form', 'application', 'report', 'document']
            if any(keyword.lower() in text.lower() for keyword in keywords):
                return True
        except ImportError:
            pass
        
        # Analyze layout structure
        edges = cv2.Canny(gray, 50, 150)
        horizontal_lines = cv2.HoughLinesP(edges, 1, np.pi/180, 100, 
                                         minLineLength=100, maxLineGap=10)
        
        return horizontal_lines is not None and len(horizontal_lines) > 5
    

    To use this solution:

    1. Install required packages:
    pip install PyMuPDF opencv-python numpy Pillow pytesseract
    
    1. For better OCR support (optional):
    # On Ubuntu/Debian
    sudo apt-get install tesseract-ocr
    # On macOS
    brew install tesseract
    
    1. Run the script:
    split_pdf("your_large_pdf.pdf", "output_directory")
    

    You can adjust the boundary detection parameters based on your specific documents:

    1. For documents with consistent headers:
    • Modify detect_document_boundary() to look for specific patterns
    • Add template matching for known document types
    1. For documents with varying layouts:
    • Implement machine learning-based classification
    • Use more sophisticated image analysis techniques

    Would you like me to explain any part in more detail or help you adapt this to your specific document types?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.