Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-4-20250514· AnthropicHugging Face LMArena Human Preference 140K

how would I create a webscraping n8n that collects all data from a website and stores it…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Mistralmistral-small-3.1-24b-instruct-2503’s answer.
  1. User

    how would I create a webscraping n8n that collects all data from a website and stores it in an airtable or google sheet, then the next day compare the prices

  2. Claude Assistant

    Here's how to create a web scraping workflow in n8n that collects price data and compares it day-over-day:

    Basic Workflow Structure

    Your n8n workflow will need these main components:

    1. Schedule Trigger - Set to run daily at a specific time
    2. HTTP Request Node - To fetch the website HTML
    3. HTML Extract Node - To parse and extract the price data
    4. Database Operations - To store and retrieve data
    5. Comparison Logic - To compare today's prices with yesterday's

    Step-by-Step Implementation

    1. Set Up the Schedule Trigger

    Add a Schedule Trigger node and configure it to run daily (e.g., every day at 9 AM). This ensures consistent data collection timing.

    2. Fetch Website Data

    Add an HTTP Request node:

    • Set the URL to your target website
    • Method: GET
    • You may need to add headers to mimic a browser request if the site blocks automated requests

    3. Extract Price Data

    Use the HTML Extract node to parse the response:

    • Configure CSS selectors or XPath to target price elements
    • Extract additional data like product names, SKUs, or descriptions
    • Structure the output as JSON for easier processing

    4. Store Data in Airtable/Google Sheets

    For Airtable:

    • Add an Airtable node
    • Configure authentication with your API key
    • Set operation to "Append"
    • Map your extracted data to table fields
    • Include a timestamp field for tracking when data was collected

    For Google Sheets:

    • Add a Google Sheets node
    • Authenticate with your Google account
    • Choose "Append" operation
    • Map data to columns
    • Add a date column for tracking

    5. Implement Price Comparison

    Create a second workflow branch that:

    • Retrieves yesterday's data using a filter (date = today - 1 day)
    • Uses a Merge node to combine today's and yesterday's data
    • Add a Function node to calculate price differences:
    const todayData = $input.all();
    const yesterdayData = $node["Get Yesterday Data"].json;
    
    return todayData.map(todayItem => {
      const yesterdayItem = yesterdayData.find(
        item => item.productId === todayItem.productId
      );
      
      if (yesterdayItem) {
        return {
          ...todayItem,
          previousPrice: yesterdayItem.price,
          priceChange: todayItem.price - yesterdayItem.price,
          percentChange: ((todayItem.price - yesterdayItem.price) / yesterdayItem.price * 100).toFixed(2)
        };
      }
      
      return {
        ...todayItem,
        previousPrice: null,
        priceChange: null,
        percentChange: null
      };
    });
    

    6. Store Comparison Results

    Add another database node to store the comparison results in a separate table/sheet with:

    • Product information
    • Current price
    • Previous price
    • Price change (absolute and percentage)
    • Date of comparison

    Important Considerations

    Website Terms of Service: Always check the website's robots.txt file and terms of service before scraping. Some sites prohibit automated data collection.

    Rate Limiting: Add wait nodes between requests if scraping multiple pages to avoid overwhelming the server.

    Error Handling: Implement IF nodes to handle scenarios where:

    • The website structure changes
    • No previous day data exists (first run)
    • Network requests fail

    Data Validation: Add nodes to validate that prices are numeric and within expected ranges before storing.

    Notifications: Consider adding email or Slack nodes to alert you when significant price changes occur.

    Alternative Approach for Dynamic Sites

    If the website uses JavaScript to load prices dynamically, you'll need to use a headless browser approach:

    • Use n8n's Puppeteer node or integrate with a service like Browserless
    • This allows JavaScript execution before extracting data

    Would you like me to elaborate on any specific part of this workflow or help you handle a particular website structure?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.