I have a PDF file that is a concatenation of several scanned documents. How to implement…
- User
I have a PDF file that is a concatenation of several scanned documents. How to implement a solution in Python that enables me to recover the original documents ?
Assistant
Here's a solution to split a PDF containing scanned documents into separate files based on visual analysis and content detection:
import cv2 import numpy as np import fitz # PyMuPDF from PIL import Image import io import os def convert_page_to_image(page): """Convert PDF page to numpy array image""" pix = page.get_pixmap() img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples) return np.array(img) def is_blank_page(image, threshold=0.95): """Detect if a page is mostly blank""" gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) white_pixels = np.sum(gray > 250) total_pixels = gray.size return (white_pixels / total_pixels) > threshold def detect_document_boundary(image): """Detect if page likely represents start of new document""" # Convert to grayscale gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) # Apply threshold _, thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU) # Look for common indicators of new document: # 1. Large text at top of page # 2. Significant white space followed by content # 3. Distinct header patterns # Simple heuristic: Check top portion of page for significant content top_portion = thresh[:int(thresh.shape[0] * 0.2), :] content_density = np.sum(top_portion < 128) / top_portion.size return content_density > 0.1 # Adjust threshold as needed def split_pdf(input_path, output_dir): """Split PDF into separate documents""" # Create output directory if it doesn't exist os.makedirs(output_dir, exist_ok=True) # Open PDF doc = fitz.open(input_path) current_doc_pages = [] doc_counter = 1 def save_current_doc(): nonlocal doc_counter if current_doc_pages: new_doc = fitz.open() for page_num in current_doc_pages: new_doc.insert_pdf(doc, from_page=page_num, to_page=page_num) output_path = os.path.join(output_dir, f'document_{doc_counter:03d}.pdf') new_doc.save(output_path) new_doc.close() doc_counter += 1 current_doc_pages.clear() for page_num in range(len(doc)): page = doc[page_num] image = convert_page_to_image(page) # Skip blank pages if is_blank_page(image): continue # If this looks like the start of a new document and we have existing pages, # save the current document if detect_document_boundary(image) and current_doc_pages: save_current_doc() current_doc_pages.append(page_num) # Save the last document save_current_doc() doc.close() def main(): input_pdf = "scanned_documents.pdf" output_dir = "split_documents" print("Starting PDF splitting process...") split_pdf(input_pdf, output_dir) print(f"Documents have been split and saved to {output_dir}/") if __name__ == "__main__": main()This solution includes several key features:
-
Page Analysis:
- Converts PDF pages to images for analysis
- Detects blank pages
- Identifies potential document boundaries
-
Document Boundary Detection:
- Looks for typical indicators of new documents
- Analyzes content density in the top portion of pages
- Can be enhanced with more sophisticated detection methods
-
Memory Efficiency:
- Processes one page at a time
- Uses PyMuPDF (fitz) for efficient PDF handling
To make this more robust, you could add these enhancements:
def enhance_boundary_detection(image): """Enhanced document boundary detection""" # Convert to grayscale gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) # Apply OCR to detect headers or title patterns try: import pytesseract text = pytesseract.image_to_string(gray) # Look for keywords indicating new document keywords = ['form', 'application', 'report', 'document'] if any(keyword.lower() in text.lower() for keyword in keywords): return True except ImportError: pass # Analyze layout structure edges = cv2.Canny(gray, 50, 150) horizontal_lines = cv2.HoughLinesP(edges, 1, np.pi/180, 100, minLineLength=100, maxLineGap=10) return horizontal_lines is not None and len(horizontal_lines) > 5To use this solution:
- Install required packages:
pip install PyMuPDF opencv-python numpy Pillow pytesseract- For better OCR support (optional):
# On Ubuntu/Debian sudo apt-get install tesseract-ocr # On macOS brew install tesseract- Run the script:
split_pdf("your_large_pdf.pdf", "output_directory")You can adjust the boundary detection parameters based on your specific documents:
- For documents with consistent headers:
- Modify
detect_document_boundary()to look for specific patterns - Add template matching for known document types
- For documents with varying layouts:
- Implement machine learning-based classification
- Use more sophisticated image analysis techniques
Would you like me to explain any part in more detail or help you adapt this to your specific document types?
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.