how would I create a webscraping n8n that collects all data from a website and stores it…
- User
how would I create a webscraping n8n that collects all data from a website and stores it in an airtable or google sheet, then the next day compare the prices
Assistant
Here's how to create a web scraping workflow in n8n that collects price data and compares it day-over-day:
Basic Workflow Structure
Your n8n workflow will need these main components:
- Schedule Trigger - Set to run daily at a specific time
- HTTP Request Node - To fetch the website HTML
- HTML Extract Node - To parse and extract the price data
- Database Operations - To store and retrieve data
- Comparison Logic - To compare today's prices with yesterday's
Step-by-Step Implementation
1. Set Up the Schedule Trigger
Add a Schedule Trigger node and configure it to run daily (e.g., every day at 9 AM). This ensures consistent data collection timing.
2. Fetch Website Data
Add an HTTP Request node:
- Set the URL to your target website
- Method: GET
- You may need to add headers to mimic a browser request if the site blocks automated requests
3. Extract Price Data
Use the HTML Extract node to parse the response:
- Configure CSS selectors or XPath to target price elements
- Extract additional data like product names, SKUs, or descriptions
- Structure the output as JSON for easier processing
4. Store Data in Airtable/Google Sheets
For Airtable:
- Add an Airtable node
- Configure authentication with your API key
- Set operation to "Append"
- Map your extracted data to table fields
- Include a timestamp field for tracking when data was collected
For Google Sheets:
- Add a Google Sheets node
- Authenticate with your Google account
- Choose "Append" operation
- Map data to columns
- Add a date column for tracking
5. Implement Price Comparison
Create a second workflow branch that:
- Retrieves yesterday's data using a filter (date = today - 1 day)
- Uses a Merge node to combine today's and yesterday's data
- Add a Function node to calculate price differences:
const todayData = $input.all(); const yesterdayData = $node["Get Yesterday Data"].json; return todayData.map(todayItem => { const yesterdayItem = yesterdayData.find( item => item.productId === todayItem.productId ); if (yesterdayItem) { return { ...todayItem, previousPrice: yesterdayItem.price, priceChange: todayItem.price - yesterdayItem.price, percentChange: ((todayItem.price - yesterdayItem.price) / yesterdayItem.price * 100).toFixed(2) }; } return { ...todayItem, previousPrice: null, priceChange: null, percentChange: null }; });6. Store Comparison Results
Add another database node to store the comparison results in a separate table/sheet with:
- Product information
- Current price
- Previous price
- Price change (absolute and percentage)
- Date of comparison
Important Considerations
Website Terms of Service: Always check the website's robots.txt file and terms of service before scraping. Some sites prohibit automated data collection.
Rate Limiting: Add wait nodes between requests if scraping multiple pages to avoid overwhelming the server.
Error Handling: Implement IF nodes to handle scenarios where:
- The website structure changes
- No previous day data exists (first run)
- Network requests fail
Data Validation: Add nodes to validate that prices are numeric and within expected ranges before storing.
Notifications: Consider adding email or Slack nodes to alert you when significant price changes occur.
Alternative Approach for Dynamic Sites
If the website uses JavaScript to load prices dynamically, you'll need to use a headless browser approach:
- Use n8n's Puppeteer node or integrate with a service like Browserless
- This allows JavaScript execution before extracting data
Would you like me to elaborate on any specific part of this workflow or help you handle a particular website structure?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.