Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-30b-a3b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

how to get puppeteer to read next js sites

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Mistralmistral-small-3.1-24b-instruct-2503’s answer.
  1. User

    how to get puppeteer to read next js sites

  2. Qwen Assistant

    To effectively use Puppeteer to scrape or interact with Next.js sites, you need to account for both server-side rendering (SSR) and client-side rendering (CSR), as Next.js often combines both. Here's a structured approach to ensure Puppeteer captures the full content:


    1. Basic Setup

    Install Puppeteer:

    npm install puppeteer
    

    2. Launch Browser and Navigate to the Page

    Use page.goto() with appropriate waitUntil options to handle dynamic content.

    const puppeteer = require('puppeteer');
    
    (async () => {
      const browser = await puppeteer.launch({ headless: true });
      const page = await browser.newPage();
    
      await page.goto('https://your-nextjs-site.com', {
        waitUntil: 'networkidle2', // Wait until no more than 2 network connections for 500ms
      });
    })();
    

    3. Wait for Dynamic Content

    Next.js sites may load content via APIs or client-side scripts. Use waitForSelector or waitForFunction to wait for elements to appear.

    Example: Wait for a Specific Element

    await page.waitForSelector('#content'); // Wait for an element with ID 'content'
    const content = await page.$eval('#content', el => el.textContent);
    console.log(content);
    

    Example: Wait for Content to Be Filled

    If the element exists but the content is dynamically populated:

    await page.waitForFunction(() => {
      return document.querySelector('#content').textContent.trim() !== '';
    });
    

    4. Handle Client-Side Navigation

    If the site uses Next.js App Router or client-side routing, ensure Puppeteer waits for navigation after interactions (e.g., clicking links).

    await page.click('a[href="/about"]');
    await page.waitForNavigation({ waitUntil: 'networkidle2' });
    

    5. Handle Loading States

    If the site shows a loading spinner or skeleton, wait for it to disappear before extracting content.

    await page.waitForSelector('.loading-spinner', { hidden: true }); // Wait until the spinner is hidden
    

    6. Adjust Timeouts

    If the site takes longer to load, increase the timeout:

    await page.goto('https://your-nextjs-site.com', {
      waitUntil: 'networkidle2',
      timeout: 60000, // 60 seconds
    });
    

    7. Extract Data from the DOM

    Use page.evaluate() to extract data directly from the DOM:

    const data = await page.evaluate(() => {
      return Array.from(document.querySelectorAll('.product')).map(el => ({
        title: el.querySelector('h2').textContent,
        price: el.querySelector('.price').textContent,
      }));
    });
    console.log(data);
    

    8. Close the Browser

    await browser.close();
    

    Full Example

    const puppeteer = require('puppeteer');
    
    (async () => {
      const browser = await puppeteer.launch({ headless: true });
      const page = await browser.newPage();
    
      await page.goto('https://your-nextjs-site.com', {
        waitUntil: 'networkidle2',
        timeout: 60000,
      });
    
      // Wait for dynamic content
      await page.waitForFunction(() => {
        return document.querySelector('#content').textContent.trim() !== '';
      });
    
      // Extract data
      const content = await page.$eval('#content', el => el.textContent);
      console.log('Page Content:', content);
    
      await browser.close();
    })();
    

    Tips

    • Use networkidle2 for sites with many API calls.
    • Avoid page.waitForTimeout() (it’s unreliable). Instead, use waitForSelector or waitForFunction.
    • Check for anti-bot measures (e.g., navigator.webdriver detection) by setting a custom user agent:
      await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36');
      

    By combining these strategies, Puppeteer can reliably interact with Next.js sites, even those with complex client-side rendering.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.