When tracing a site (built with Shopify) via puppeteer it appears that JS is not…
- User
When tracing a site (built with Shopify) via puppeteer it appears that JS is not executed. JS execution is enabled and works on other sites. Why does it not work on this one and how to make it work.
Screenshots lack elements added with JS and the trace doesn't include main thread JS activity.
- Assistant
This is likely due to Shopify's bot detection mechanisms. Shopify sites often implement various anti-bot measures that can detect and block automated browsers like Puppeteer. Here are the most common causes and solutions:
Common Causes
- Headless Detection: Shopify can detect headless browsers through various JavaScript checks
- Browser Fingerprinting: Automated browsers have distinct fingerprints that differ from regular browsers
- User Agent Detection: Default Puppeteer user agents are easily identifiable
- WebDriver Detection: Sites can check for
navigator.webdriverproperty
Solutions to Try
1. Run in Headful Mode
const browser = await puppeteer.launch({ headless: false, args: ['--no-sandbox', '--disable-setuid-sandbox'] });2. Use Stealth Plugin
Install and use puppeteer-extra with the stealth plugin:
const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin()); const browser = await puppeteer.launch({ headless: true });3. Set Realistic Viewport and User Agent
await page.setViewport({ width: 1920, height: 1080 }); await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36');4. Add Delays and Wait Strategies
// Wait for network to be idle await page.goto(url, { waitUntil: 'networkidle2' }); // Or wait for specific elements await page.waitForSelector('.your-js-element', { timeout: 30000 }); // Add realistic delays await page.waitForTimeout(2000);5. Check for Specific Blocking
// Log console messages to see if there are errors page.on('console', msg => console.log('PAGE LOG:', msg.text())); // Check if specific scripts are being blocked const scripts = await page.evaluate(() => { return Array.from(document.scripts).map(s => s.src); }); console.log('Loaded scripts:', scripts);6. Try Different Launch Arguments
const browser = await puppeteer.launch({ headless: false, args: [ '--disable-blink-features=AutomationControlled', '--disable-features=IsolateOrigins,site-per-process', '--no-sandbox', '--disable-setuid-sandbox', '--disable-dev-shm-usage', '--disable-accelerated-2d-canvas', '--no-first-run', '--no-zygote', '--single-process', '--disable-gpu' ] });If these don't work, the site might be using more sophisticated detection methods. In that case, you might need to use tools like Playwright with its more advanced stealth capabilities, or consider using a service that specializes in bypassing such protections for legitimate use cases.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.