Skip to content
The Chat Archive
claude-opus-4-20250514-thinking-16k· Anthropic LMArena Human Preference 140K

When tracing a site (built with Shopify) via puppeteer it appears that JS is not…

  1. User

    When tracing a site (built with Shopify) via puppeteer it appears that JS is not executed. JS execution is enabled and works on other sites. Why does it not work on this one and how to make it work.

    Screenshots lack elements added with JS and the trace doesn't include main thread JS activity.

  2. Assistant

    This is likely due to Shopify's bot detection mechanisms. Shopify sites often implement various anti-bot measures that can detect and block automated browsers like Puppeteer. Here are the most common causes and solutions:

    Common Causes

    1. Headless Detection: Shopify can detect headless browsers through various JavaScript checks
    2. Browser Fingerprinting: Automated browsers have distinct fingerprints that differ from regular browsers
    3. User Agent Detection: Default Puppeteer user agents are easily identifiable
    4. WebDriver Detection: Sites can check for navigator.webdriver property

    Solutions to Try

    1. Run in Headful Mode

    const browser = await puppeteer.launch({
      headless: false,
      args: ['--no-sandbox', '--disable-setuid-sandbox']
    });
    

    2. Use Stealth Plugin

    Install and use puppeteer-extra with the stealth plugin:

    const puppeteer = require('puppeteer-extra');
    const StealthPlugin = require('puppeteer-extra-plugin-stealth');
    puppeteer.use(StealthPlugin());
    
    const browser = await puppeteer.launch({ headless: true });
    

    3. Set Realistic Viewport and User Agent

    await page.setViewport({ width: 1920, height: 1080 });
    await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36');
    

    4. Add Delays and Wait Strategies

    // Wait for network to be idle
    await page.goto(url, { waitUntil: 'networkidle2' });
    
    // Or wait for specific elements
    await page.waitForSelector('.your-js-element', { timeout: 30000 });
    
    // Add realistic delays
    await page.waitForTimeout(2000);
    

    5. Check for Specific Blocking

    // Log console messages to see if there are errors
    page.on('console', msg => console.log('PAGE LOG:', msg.text()));
    
    // Check if specific scripts are being blocked
    const scripts = await page.evaluate(() => {
      return Array.from(document.scripts).map(s => s.src);
    });
    console.log('Loaded scripts:', scripts);
    

    6. Try Different Launch Arguments

    const browser = await puppeteer.launch({
      headless: false,
      args: [
        '--disable-blink-features=AutomationControlled',
        '--disable-features=IsolateOrigins,site-per-process',
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-dev-shm-usage',
        '--disable-accelerated-2d-canvas',
        '--no-first-run',
        '--no-zygote',
        '--single-process',
        '--disable-gpu'
      ]
    });
    

    If these don't work, the site might be using more sophisticated detection methods. In that case, you might need to use tools like Playwright with its more advanced stealth capabilities, or consider using a service that specializes in bypassing such protections for legitimate use cases.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.