[code] Currently this converts each article to PDF with a two-column layout and then…
- User
from pathlib import Path from PyPDF2 import PdfMerger import os import logging from weasyprint import HTML, CSS logger = logging.getLogger(__name__) def write_pdf_weasy(html_path, pdf_path): # Read HTML content with open(html_path, encoding='utf-8') as f: html_content = f.read() # Define CSS for two-column layout and pagination css = CSS(string=""" @page { size: A4 landscape; margin: 20px; } body { font-size: 7pt; font-family: sans-serif; column-count: 2; column-gap: 20px; margin: 0; padding: 20px; } p, h1, h2, h3, blockquote { break-inside: avoid; page-break-inside: avoid; } h1 { font-size: 16pt; } figure img { max-width: 100%; max-height: 200px; height: auto; display: block; margin: 0 auto; } figure { break-inside: avoid; margin: 1em 0; } """) # Generate PDF HTML(string=html_content).write_pdf(pdf_path, stylesheets=[css]) def merge_pdfs(pdf_files, output_path): """Merge multiple PDF files into one""" merger = PdfMerger() for pdf in pdf_files: merger.append(str(pdf)) merger.write(output_path) logger.debug("Wrote combined PDF", extra={"output_path": output_path}) merger.close() return output_path async def _combine(directory: Path, pattern: str = "*.html"): html_files = list(directory.glob(pattern)) pdf_files = [html.with_name(html.stem + ".pdf") for html in html_files] for html, pdf in zip(html_files, pdf_files): write_pdf_weasy(str(html), pdf) # Remember that the parent's name is the feed id # and the child is the datetime of its latest entry # Remember that we assume the directory name is the feed id outfile_name = f"{directory.parent.name}_{directory.name}_combined.pdf" # Join the outfile_name with the directory outfile_path = directory / outfile_name merge_pdfs(pdf_files, outfile_path) for pdf in pdf_files: try: os.remove(pdf) logger.debug("Deleted pdf", extra={"filepath": pdf}) except Exception: logger.error("Failed to delete pdf", extra={"filepath": pdf}, exc_info=True)Currently this converts each article to PDF with a two-column layout and then concatenates the PDFs. How might we instead concatenate so that articles can flow smoothly? For example one article might finish in a left column and another start in the right column.
- Assistant
Okay, to achieve a smooth flow where articles can continue from one column to the next, or one page to the next, without hard breaks, you need to:
- Combine HTML Content: Instead of generating a PDF for each HTML and then merging PDFs (which just concatenates pages), you need to merge the HTML content of all articles into a single, large HTML document.
- Render Single HTML to PDF: Pass this combined HTML document to WeasyPrint once. WeasyPrint will then lay out the entire content according to your CSS, allowing text to flow across columns and pages naturally.
Here's how you can modify your code. This will involve:
- Reading each HTML file.
- Using a library like
BeautifulSoupto parse the HTML and extract the content within the<body>tags. This is more robust than simple string concatenation of full HTML files. - Resolving relative URLs (like for images) within each HTML fragment to absolute URLs before combining, so they still work in the context of the new combined document.
- Concatenating these body contents, possibly with a separator (like
<hr>) between articles. - Creating a new, complete HTML string with this combined body.
- Passing this single HTML string to a modified
write_pdf_weasyor directly to WeasyPrint. - Removing the
merge_pdfsfunction and the logic for creating/deleting intermediate PDFs.
You'll need to install
BeautifulSoup4andlxml(optional but recommended parser for BeautifulSoup):pip install beautifulsoup4 lxmlHere's the revised code:
from pathlib import Path from PyPDF2 import PdfMerger # No longer needed for the primary goal, but kept if other uses exist import os import logging from weasyprint import HTML, CSS from bs4 import BeautifulSoup # For parsing HTML from urllib.parse import urljoin, urlparse # For resolving relative URLs import asyncio # For running blocking IO/CPU tasks in async context import sys # To check for lxml logger = logging.getLogger(__name__) # Helper function to run blocking code in a thread pool for async compatibility async def run_in_executor(func, *args): loop = asyncio.get_event_loop() return await loop.run_in_executor(None, func, *args) def generate_pdf_from_html_string(html_string, pdf_path, custom_css_string=None): """ Generates a PDF from an HTML string using WeasyPrint with a two-column layout. """ # Define CSS for two-column layout and pagination # Merged the CSS from the original write_pdf_weasy and added for separator default_css_string = """ @page { size: A4 landscape; margin: 20px; } body { font-size: 7pt; font-family: sans-serif; column-count: 2; column-gap: 20px; margin: 0; /* Body margin is often controlled by @page margin */ padding: 0; /* Padding on body within @page margins */ } p, h1, h2, h3, blockquote, ul, ol, dl { /* Added list types */ break-inside: avoid; page-break-inside: avoid; /* Alias for break-inside */ } h1 { font-size: 16pt; /* To avoid a heading at the very end of a column if possible */ break-after: avoid-column; } h2 { font-size: 12pt; break-after: avoid-column; } h3 { font-size: 10pt; break-after: avoid-column; } figure img { max-width: 100%; /* relative to its container (a column) */ max-height: 300px; /* Increased a bit, adjust as needed */ height: auto; display: block; margin: 10px auto; /* some vertical margin */ } figure { break-inside: avoid; margin: 1em 0; padding: 0; /* Ensure figure doesn't add unexpected padding */ } /* Class for a separator between combined articles */ .article-separator { border: 0; border-top: 1px solid #ccc; margin: 1.5em 5%; /* Vertical space and slight horizontal inset */ break-inside: avoid; /* Don't let the HR itself break */ /* We want flow, so avoid break-before: column unless desired */ } /* Basic table styling */ table { width: 100%; border-collapse: collapse; margin-bottom: 1em; break-inside: avoid; } th, td { border: 1px solid #ddd; padding: 4px; text-align: left; } th { background-color: #f2f2f2; } """ css_to_use = custom_css_string if custom_css_string else default_css_string css = CSS(string=css_to_use) # Generate PDF # The base_url is important if the HTML string contains relative paths # that were NOT resolved to absolute paths during HTML combination. # However, our combination step below aims to make paths absolute. HTML(string=html_string).write_pdf(pdf_path, stylesheets=[css]) logger.info("Generated PDF", extra={"path": pdf_path}) def _combine_html_content_sync(html_file_paths, feed_id_for_title, date_for_title): """ Synchronous helper to combine HTML content from multiple files. Resolves relative URLs to absolute ones. """ all_body_elements_html = [] doc_title = f"{feed_id_for_title}_{date_for_title}_combined" # Determine preferred parser parser = 'lxml' if 'lxml' in sys.modules else 'html.parser' for i, html_file_path_obj in enumerate(html_file_paths): html_file_path_str = str(html_file_path_obj) # base_uri for resolving relative links within this specific HTML file base_uri = html_file_path_obj.resolve().as_uri() try: with open(html_file_path_str, 'r', encoding='utf-8') as f: soup = BeautifulSoup(f, parser) if soup.body: # Resolve relative URLs for images, links, stylesheets, scripts # Make sure to handle various ways relative paths can be specified. for tag in soup.body.find_all(True, src=True): # Finds any tag with an src attribute src = tag['src'] if not urlparse(src).scheme and not src.startswith('data:'): tag['src'] = urljoin(base_uri, src) for tag in soup.body.find_all(True, href=True): # Finds any tag with an href attribute href = tag['href'] # Don't modify internal page anchors (#fragment) if not href.startswith('#') and not urlparse(href).scheme and not href.startswith('data:'): tag['href'] = urljoin(base_uri, href) # Extract children of <body> as string body_content_html = ''.join(str(content_item) for content_item in soup.body.contents) all_body_elements_html.append(body_content_html) # Add a separator, but not after the very last article if i < len(html_file_paths) - 1: all_body_elements_html.append('<hr class="article-separator" />') else: logger.warning(f"No <body> tag found in {html_file_path_str}") except Exception as e: logger.error(f"Failed to process HTML file {html_file_path_str}: {e}", exc_info=True) all_body_elements_html.append(f"<p>Error processing article from {html_file_path_obj.name}: {e}</p>") if i < len(html_file_paths) - 1: all_body_elements_html.append('<hr class="article-separator" />') combined_body_html = "\n".join(all_body_elements_html) # Construct a full HTML document string final_html_str = f""" <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>{doc_title}</title> </head> <body> {combined_body_html} </body> </html> """ return final_html_str async def combine_articles_to_single_pdf(directory: Path, pattern: str = "*.html"): html_files = sorted(list(directory.glob(pattern))) # Sort for consistent order if not html_files: logger.warning("No HTML files found to combine.", extra={"directory": directory, "pattern": pattern}) return # Prepare arguments for the synchronous HTML combination part feed_id = directory.parent.name datetime_str = directory.name # Assuming this is the datetime string # Run the blocking HTML processing in an executor combined_html_string = await run_in_executor( _combine_html_content_sync, html_files, feed_id, datetime_str ) # Define output PDF path outfile_name = f"{feed_id}_{datetime_str}_combined_flowing.pdf" outfile_path = directory / outfile_name # Run the blocking PDF generation in an executor # You can pass a custom CSS string to generate_pdf_from_html_string if needed await run_in_executor( generate_pdf_from_html_string, combined_html_string, str(outfile_path) # WeasyPrint expects path as string or Path object ) # No intermediate PDFs to delete with this approach logger.info("Successfully created combined PDF with flowing content.", extra={"output_path": outfile_path}) return outfile_path # Example Usage (assuming you call this from an async context): async def main_example(): logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s') # Create dummy HTML files for testing # Structure: test_feed_id / test_datetime_dir / article_N.html test_feed_id = "my_feed" test_datetime_dir = "20231027100000" base_dir = Path("temp_test_data") test_dir = base_dir / test_feed_id / test_datetime_dir test_dir.mkdir(parents=True, exist_ok=True) html_content_1 = """ <html><head><title>Article 1</title></head> <body> <h1>Article One: The Beginning</h1> <p>This is the first paragraph of the first article. It should flow nicely. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.</p> <figure><img src="https://via.placeholder.com/300x100.png?text=Img+Art+1" alt="Placeholder Image 1"></figure> <p>Another paragraph in article one. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.</p> </body></html> """ html_content_2 = """ <html><head><title>Article 2</title></head> <body> <h1>Article Two: The Sequel</h1> <p>This is the start of the second article. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident.</p> <blockquote>A blockquote here. This should ideally not break across columns.</blockquote> <p>And more content for article two. Sunt in culpa qui officia deserunt mollit anim id est laborum. This article might start on the right column if the first one ends on the left.</p> <figure><img src="https://via.placeholder.com/250x150.png?text=Img+Art+2" alt="Placeholder Image 2"></figure> </body></html> """ html_content_3 = """ <html><head><title>Article 3</title></head> <body> <h1>Article Three: The Grand Finale with a Table</h1> <p>Welcome to the third article, which will test how tables are handled in this flowing layout.</p> <table> <thead><tr><th>Header 1</th><th>Header 2</th><th>Header 3</th></tr></thead> <tbody> <tr><td>Data 1A</td><td>Data 1B</td><td>Data 1C</td></tr> <tr><td>Data 2A</td><td>Data 2B</td><td>Data 2C</td></tr> <tr><td>Data 3A</td><td>Data 3B</td><td>Data 3C. This cell might contain a bit more text to see how it wraps and affects column flow.</td></tr> </tbody> </table> <p>Some text after the table to see how the flow continues. Hopefully, the table and its content are kept together as much as possible without causing awkward breaks.</p> </body></html> """ with open(test_dir / "article_1.html", "w", encoding="utf-8") as f: f.write(html_content_1) with open(test_dir / "article_2.html", "w", encoding="utf-8") as f: f.write(html_content_2) with open(test_dir / "article_3.html", "w", encoding="utf-8") as f: f.write(html_content_3) output_pdf = await combine_articles_to_single_pdf(test_dir) if output_pdf: print(f"Generated combined PDF: {output_pdf}") # Cleanup dummy files (optional) # for f in test_dir.glob("*.html"): os.remove(f) # if output_pdf and os.path.exists(output_pdf): os.remove(output_pdf) # os.rmdir(test_dir) # os.rmdir(test_dir.parent) # os.rmdir(base_dir) if __name__ == "__main__": # If running this script directly: asyncio.run(main_example())Key changes and explanations:
-
BeautifulSoupand URL Resolution:- The
_combine_html_content_syncfunction now takes a list of HTML file paths. - For each file, it reads the content, parses it with
BeautifulSoup. - Crucially, it finds tags with
srcorhrefattributes (like<img>,<a>,<link>,<script>) and converts any relative URLs to absolute URLs usingurljoinand the original file's path as the base. This ensures images and links still work. - It extracts the content of the
<body>tag. - An
<hr class="article-separator" />is added between the content of different articles.
- The
-
Single HTML String:
- All extracted (and now URL-resolved) body contents are joined into one long HTML string, wrapped in a basic
<html><head><body>...</body></head></html>structure.
- All extracted (and now URL-resolved) body contents are joined into one long HTML string, wrapped in a basic
-
generate_pdf_from_html_string:- This function (modified from your
write_pdf_weasy) now takes an HTML string instead of a file path. - The CSS is defined here. I've added a style for
.article-separatorand some basic table styling. I also tweakedh1,h2,h3to usebreak-after: avoid-column;to try and prevent headings from being the last item in a column.
- This function (modified from your
-
combine_articles_to_single_pdf(was_combine):- This is the main async orchestrator.
- It gets the list of HTML files.
- It calls
_combine_html_content_sync(which is blocking I/O and CPU work) usingawait run_in_executorto keep the async event loop free. - It then calls
generate_pdf_from_html_string(also blocking WeasyPrint work) usingawait run_in_executor. - The logic for merging PDFs and deleting intermediate PDFs is removed because it's no longer needed.
-
Async Correctness (
run_in_executor):- File I/O (reading HTMLs), HTML parsing (BeautifulSoup), and PDF generation (WeasyPrint) are blocking operations. When working in an
asyncenvironment, these should be run in a separate thread pool usingloop.run_in_executor(None, blocking_function, arg1, arg2)to prevent them from freezing the asyncio event loop. I've added a helperrun_in_executorand used it.
- File I/O (reading HTMLs), HTML parsing (BeautifulSoup), and PDF generation (WeasyPrint) are blocking operations. When working in an
-
CSS for Flow:
- The existing CSS (
break-inside: avoidfor paragraphs, headings, etc.) is good for preventing individual elements from breaking across columns. - The new
.article-separatorclass styles the<hr>tag.break-inside: avoid;on the separator itself helps prevent the line from splitting. - The content will naturally flow. If an article's text ends mid-way down the left column, the next article (after the separator) will start immediately in that same left column, or at the top of the right column if the left is full, or on the next page if the current page is full.
- The existing CSS (
This revised approach directly addresses the requirement for smooth content flow by treating all articles as part of a single document for rendering.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.