1) Project Summary Build an autonomous web scraper using Playwright that extracts Publix…
- User
- Project Summary Build an autonomous web scraper using Playwright that extracts Publix BOGO weekly ad data and exports it to an Excel file. Goal: Extract Title, Badge, Description, Save Up To, Valid Dates, and Product Image for each BOGO item.
- Problem Summary "Save Up To" column always comes out blank. Selector for "Save Up To" (class .p-price__savings) not consistently available at extraction time. All other fields extract successfully. Code times out or fails when waiting for .p-price__savings selector.
- Things Attempted So Far (bullet list)
- ✅ Initial version extracting Title, Badge, Description, Valid Dates, Image — WORKS.
- ❌ Added extraction for "Save Up To" using .p-price__savings — FAILS, column blank.
- ❌ Global wait_for_selector('.p-price__savings') — FAILS, timeout.
- ❌ Per-card querySelector('.p-price__savings') — FAILS, blank.
- ❌ Extended wait_for_timeout() after page load — FAILS, still blank.
- ✅ Image downloading works perfectly.
- ❌ Tried Playwright EXE packaging — ABANDONED.
- Ideas to Potentially Fix (bullet list)
- Add longer wait_for_timeout after page load and after each scroll to ensure full rendering.
- Use MutationObserver to watch for dynamic loading of savings elements.
- Try scraping from possible backend API instead of DOM.
- Use Playwright .is_visible() or .inner_text() dynamically rather than querySelector chaining.
- The Latest Full Code Version (self-contained full code, including all imports) import asyncio import pandas as pd import xlsxwriter import requests from io import BytesIO from PIL import Image import os from playwright.async_api import async_playwright
PUBLIX_URL = "https://www.publix.com/savings/weekly-ad/bogo" OUTPUT_DIR = "output" os.makedirs(OUTPUT_DIR, exist_ok=True)
def download_image(url, idx): try: response = requests.get(url, timeout=10) response.raise_for_status() img = Image.open(BytesIO(response.content)) img_path = os.path.join(OUTPUT_DIR, f"temp_img_{idx}.png") img.save(img_path) return img_path except: return None
def save_to_excel(items): filename = os.path.join(OUTPUT_DIR, "publix_bogo_deals.xlsx") workbook = xlsxwriter.Workbook(filename) worksheet = workbook.add_worksheet() headers = ["Title", "Badge", "Description", "Save Up To", "Valid Dates"] for col, header in enumerate(headers): worksheet.write(0, col, header) worksheet.set_column("A:E", 35) worksheet.set_column("F:F", 20) for idx, item in enumerate(items): row = idx + 1 worksheet.write(row, 0, item.get("title")) worksheet.write(row, 1, item.get("badge")) worksheet.write(row, 2, item.get("description")) worksheet.write(row, 3, item.get("save_up_to")) worksheet.write(row, 4, item.get("valid")) img_path = download_image(item.get("img"), idx) if img_path: worksheet.set_row(row, 100) worksheet.insert_image(row, 5, img_path, {"x_scale": 0.5, "y_scale": 0.5}) workbook.close()
async def run_scraper(): async with async_playwright() as p: browser = await p.chromium.launch(headless=False) context = await browser.new_context() page = await context.new_page() await page.goto(PUBLIX_URL) await page.wait_for_timeout(8000)
for _ in range(10): await page.mouse.wheel(0, 1500) await page.wait_for_timeout(1000) await page.wait_for_selector('.p-card', timeout=8000) await page.wait_for_timeout(3000) extraction_script = """ () => Array.from(document.querySelectorAll('.p-card')).map(card => ({ title: card.querySelector('.title-wrapper')?.innerText, img: card.querySelector('img')?.src, badge: card.querySelector('.p-savings-badge__text')?.innerText, description: card.querySelector('.p-savings-badge__description')?.innerText, save_up_to: card.querySelector('.p-price__savings')?.innerText || '', valid: card.querySelector('.valid-dates')?.innerText })) """ items = await page.evaluate(extraction_script) print(f"Extracted {len(items)} items.") save_to_excel(items) await browser.close()if name == "main": asyncio.run(run_scraper())
Assistant
To resolve the issue of the "Save Up To" field being blank in your Publix BOGO ad scraper, we need to address the dynamic loading of the
.p-price__savingselements. The root cause is that the savings information is not immediately available in the DOM when the initial extraction script runs. Here's a structured solution:
✅ Step-by-Step Fixes
1. Ensure Elements Are Fully Rendered
- The
.p-price__savingselements may be dynamically loaded after the page scrolls. Increase the wait times and ensure all content is fully rendered.
# Increase wait times after scrolling and before extraction await page.wait_for_timeout(5000) # Wait longer after scrolling2. Wait for
.p-price__savingsExplicitly- Use
page.wait_for_selectorwith a longer timeout to ensure the savings elements are present before extraction.
await page.wait_for_selector('.p-price__savings', timeout=15000) # 15 seconds3. Refactor Extraction Script to Handle Dynamic Content
- Update the script to wait for each card's savings element individually, ensuring it is visible and rendered.
extraction_script = """ () => { const items = []; const cards = document.querySelectorAll('.p-card'); cards.forEach(card => { const title = card.querySelector('.title-wrapper')?.innerText || ''; const img = card.querySelector('img')?.src || ''; const badge = card.querySelector('.p-savings-badge__text')?.innerText || ''; const description = card.querySelector('.p-savings-badge__description')?.innerText || ''; const valid = card.querySelector('.valid-dates')?.innerText || ''; const saveUpTo = card.querySelector('.p-price__savings')?.innerText || ''; items.push({ title, img, badge, description, saveUpTo, valid }); }); return items; } """4. Add Visibility Checks
- Ensure the
.p-price__savingselement is visible before extracting text. This avoids cases where the element exists in the DOM but is hidden.
# In the extraction script, check if the element is visible const saveUpTo = card.querySelector('.p-price__savings')?.getBoundingClientRect().height > 0 ? card.querySelector('.p-price__savings')?.innerText : '';5. Simulate Hover to Trigger Dynamic Content
- If the savings info appears on hover, simulate mouse movement over each card.
for card in cards: await card.hover() await page.wait_for_timeout(500) # Allow time for content to load6. Verify DOM Structure
- Use browser dev tools to confirm the exact location of
.p-price__savings. If it's not a direct child of.p-card, adjust the selector accordingly.
🛠️ Updated Code Example
Here’s the revised code incorporating the above fixes:
import asyncio import pandas as pd import xlsxwriter import requests from io import BytesIO from PIL import Image import os from playwright.async_api import async_playwright PUBLIX_URL = "https://www.publix.com/savings/weekly-ad/bogo" OUTPUT_DIR = "output" os.makedirs(OUTPUT_DIR, exist_ok=True) def download_image(url, idx): try: response = requests.get(url, timeout=10) response.raise_for_status() img = Image.open(BytesIO(response.content)) img_path = os.path.join(OUTPUT_DIR, f"temp_img_{idx}.png") img.save(img_path) return img_path except: return None def save_to_excel(items): filename = os.path.join(OUTPUT_DIR, "publix_bogo_deals.xlsx") workbook = xlsxwriter.Workbook(filename) worksheet = workbook.add_worksheet() headers = ["Title", "Badge", "Description", "Save Up To", "Valid Dates"] for col, header in enumerate(headers): worksheet.write(0, col, header) worksheet.set_column("A:E", 35) worksheet.set_column("F:F", 20) for idx, item in enumerate(items): row = idx + 1 worksheet.write(row, 0, item.get("title")) worksheet.write(row, 1, item.get("badge")) worksheet.write(row, 2, item.get("description")) worksheet.write(row, 3, item.get("save_up_to")) worksheet.write(row, 4, item.get("valid")) img_path = download_image(item.get("img"), idx) if img_path: worksheet.set_row(row, 100) worksheet.insert_image(row, 5, img_path, {"x_scale": 0.5, "y_scale": 0.5}) workbook.close() async def run_scraper(): async with async_playwright() as p: browser = await p.chromium.launch(headless=False) context = await browser.new_context() page = await context.new_page() await page.goto(PUBLIX_URL) await page.wait_for_timeout(5000) # Scroll to load all cards for _ in range(10): await page.mouse.wheel(0, 1500) await page.wait_for_timeout(1000) # Wait for all cards to be present await page.wait_for_selector('.p-card', timeout=10000) await page.wait_for_timeout(3000) # Wait for savings elements to be present await page.wait_for_selector('.p-price__savings', timeout=15000) # Extract data with dynamic visibility check extraction_script = """ () => { const items = []; const cards = document.querySelectorAll('.p-card'); cards.forEach(card => { const title = card.querySelector('.title-wrapper')?.innerText || ''; const img = card.querySelector('img')?.src || ''; const badge = card.querySelector('.p-savings-badge__text')?.innerText || ''; const description = card.querySelector('.p-savings-badge__description')?.innerText || ''; const valid = card.querySelector('.valid-dates')?.innerText || ''; const saveUpTo = card.querySelector('.p-price__savings')?.getBoundingClientRect().height > 0 ? card.querySelector('.p-price__savings')?.innerText : ''; items.push({ title, img, badge, description, saveUpTo, valid }); }); return items; } """ items = await page.evaluate(extraction_script) print(f"Extracted {len(items)} items.") save_to_excel(items) await browser.close() if __name__ == "__main__": asyncio.run(run_scraper())
🔍 Additional Tips
- Inspect the DOM: Use browser dev tools to confirm the structure of
.p-price__savingsand ensure it's not inside an iframe or shadow DOM. - Check for API Calls: If the data is fetched via an API, reverse-engineer the endpoint and scrape it directly for more reliability.
- Error Handling: Add try-except blocks to handle edge cases where elements might still be missing.
By implementing these changes, the "Save Up To" field should now populate correctly, ensuring a complete and accurate dataset.
- The
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.