Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

I need a fairly efficient Python program used to search products on certain websites,…

  1. User

    I need a fairly efficient Python program used to search products on certain websites, primarily laptops, computer components and phones. The end result would be a basic website where the user can search for a product name, then filter the results. The results come from performing searches on the best technology related websites from Spain, which are PcComponentes, Coolmod, Idealo and MediaMarkt. The program should make use of already existing scraping systems in order to find data like the search bar, product names, prices, filters, etc, on an organized and easy manner, and it should try to stay, at least, a bit undetected in order to extract this small amount of data for the user.

  2. Assistant

    Creating a Python program to scrape product data from multiple Spanish e-commerce websites (PcComponentes, Coolmod, Idealo, MediaMarkt) and display it on a basic web interface involves several steps:

    1. Web Scraping: Extracting data from the target websites.
    2. Data Storage: Storing the scraped data in a database.
    3. Web Interface: Building a user-friendly website to search and filter products.

    Below is a comprehensive guide to building this system using Python, leveraging existing libraries and frameworks to ensure efficiency and maintainability.


    1. Web Scraping

    Tools and Libraries

    • Scrapy: A powerful and efficient web scraping framework.
    • BeautifulSoup: For parsing HTML documents (alternative to Scrapy selectors).
    • Requests: To make HTTP requests.
    • Selenium: For websites that heavily rely on JavaScript (optional).
    • Scrapy-UserAgents and Scrapy-Proxies: To rotate user agents and proxies for staying undetected.

    Setup

    First, install the necessary libraries:

    pip install scrapy scrapy-user-agents scrapy-proxies beautifulsoup4 requests selenium
    

    Scrapy Project Structure

    Create a Scrapy project:

    scrapy startproject product_scraper
    cd product_scraper
    

    Example Spider for PcComponentes

    Each website will require a separate spider due to differing structures. Below is an example spider for PcComponentes.

    # product_scraper/spiders/pccomponentes_spider.py
    
    import scrapy
    from scrapy import Request
    from urllib.parse import urlencode
    
    class PcComponentesSpider(scrapy.Spider):
        name = "pccomponentes"
        allowed_domains = ["www.pccomponentes.com"]
    
        def __init__(self, search_term='', *args, **kwargs):
            super(PcComponentesSpider, self).__init__(*args, **kwargs)
            self.search_term = search_term
            self.start_urls = [f"https://www.pccomponentes.com/busqueda/{self.search_term}"]
    
        def parse(self, response):
            products = response.css('div.product-wrapper')
            for product in products:
                yield {
                    'name': product.css('h2.product-name a::text').get().strip(),
                    'price': product.css('span.price::text').get().strip(),
                    'url': product.css('h2.product-name a::attr(href)').get(),
                    'image': product.css('img::attr(src)').get()
                }
    
            # Pagination
            next_page = response.css('a.next::attr(href)').get()
            if next_page:
                yield Request(url=next_page, callback=self.parse)
    

    Handling Other Websites

    Similarly, create spiders for Coolmod, Idealo, and MediaMarkt by analyzing their HTML structure and adjusting the selectors accordingly.

    Middleware for Staying Undetected

    Edit settings.py to include middleware for rotating user agents and proxies.

    # product_scraper/settings.py
    
    DOWNLOADER_MIDDLEWARES = {
        'scrapy_user_agents.middleware.RandomUserAgentMiddleware': 400,
        'scrapy_proxies.RandomProxy': 100,
        'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
    }
    
    # User Agents
    USER_AGENTS = [
        # List of user agents
    ]
    
    # Proxies
    PROXY_LIST = '/path/to/proxy/list.txt'
    PROXY_MODE = 0
    

    Ensure you have a list of proxies in proxy/list.txt. Services like ScraperAPI or ProxyScrape can provide proxies.

    Running the Spider

    You can run the spider with a search term like this:

    scrapy crawl pccomponentes -a search_term=laptop -o data/pccomponentes.json
    

    Automating Scraping for All Websites

    Create separate spiders and run them sequentially or concurrently using a master script or Scrapy’s CrawlSpider.


    2. Data Storage

    Use SQLite for simplicity or PostgreSQL for scalability. Below is an example using SQLite with SQLAlchemy.

    Install SQLAlchemy

    pip install sqlalchemy
    

    Define Models

    # product_scraper/models.py
    
    from sqlalchemy import create_engine, Column, String, Float, Integer
    from sqlalchemy.ext.declarative import declarative_base
    from sqlalchemy.orm import sessionmaker
    
    Base = declarative_base()
    
    class Product(Base):
        __tablename__ = 'products'
        id = Column(Integer, primary_key=True)
        website = Column(String)
        name = Column(String)
        price = Column(Float)
        url = Column(String)
        image = Column(String)
    
    # Database setup
    engine = create_engine('sqlite:///products.db')
    Base.metadata.create_all(engine)
    Session = sessionmaker(bind=engine)
    

    Storing Scraped Data

    Modify the spiders to store data into the database instead of outputting to JSON.

    # product_scraper/spiders/pccomponentes_spider.py
    
    import scrapy
    from product_scraper.models import Product, Session
    
    class PcComponentesSpider(scrapy.Spider):
        name = "pccomponentes"
        allowed_domains = ["www.pccomponentes.com"]
    
        def __init__(self, search_term='', *args, **kwargs):
            super(PcComponentesSpider, self).__init__(*args, **kwargs)
            self.search_term = search_term
            self.start_urls = [f"https://www.pccomponentes.com/busqueda/{self.search_term}"]
            self.session = Session()
    
        def parse(self, response):
            products = response.css('div.product-wrapper')
            for product in products:
                name = product.css('h2.product-name a::text').get().strip()
                price_text = product.css('span.price::text').get().strip()
                price = float(price_text.replace('€', '').replace(',', '.'))
                url = product.css('h2.product-name a::attr(href)').get()
                image = product.css('img::attr(src)').get()
                
                new_product = Product(
                    website='PcComponentes',
                    name=name,
                    price=price,
                    url=url,
                    image=image
                )
                self.session.add(new_product)
            
            self.session.commit()
    
            # Pagination
            next_page = response.css('a.next::attr(href)').get()
            if next_page:
                yield response.follow(next_page, callback=self.parse)
    
        def closed(self, reason):
            self.session.close()
    

    Repeat similar modifications for other spiders, ensuring the website field is appropriately set.


    3. Building the Web Interface

    Use Flask to create a simple web application where users can search and filter products.

    Install Flask and SQLAlchemy

    pip install flask sqlalchemy
    

    Create the Flask App

    # app.py
    
    from flask import Flask, render_template, request
    from sqlalchemy import create_engine, or_
    from sqlalchemy.orm import sessionmaker
    from product_scraper.models import Product, Base
    
    app = Flask(__name__)
    
    # Database setup
    engine = create_engine('sqlite:///products.db')
    Base.metadata.bind = engine
    DBSession = sessionmaker(bind=engine)
    session = DBSession()
    
    @app.route('/', methods=['GET', 'POST'])
    def index():
        if request.method == 'POST':
            search_term = request.form.get('search')
            filter_min = request.form.get('min_price', type=float)
            filter_max = request.form.get('max_price', type=float)
            
            query = session.query(Product).filter(Product.name.ilike(f'%{search_term}%'))
            
            if filter_min is not None:
                query = query.filter(Product.price >= filter_min)
            if filter_max is not None:
                query = query.filter(Product.price <= filter_max)
            
            results = query.all()
            return render_template('index.html', products=results, search=search_term)
        
        return render_template('index.html', products=[], search='')
    
    if __name__ == '__main__':
        app.run(debug=True)
    

    Create the HTML Template

    Create a templates folder in the same directory as app.py and add index.html:

    <!-- templates/index.html -->
    
    <!DOCTYPE html>
    <html>
    <head>
        <title>Product Search</title>
        <style>
            /* Basic styling */
            body { font-family: Arial, sans-serif; }
            .product { border: 1px solid #ddd; padding: 10px; margin: 10px; }
            .product img { max-width: 150px; }
        </style>
    </head>
    <body>
        <h1>Product Search</h1>
        <form method="POST">
            <input type="text" name="search" placeholder="Product name" value="{{ search }}" required>
            <input type="number" step="0.01" name="min_price" placeholder="Min Price">
            <input type="number" step="0.01" name="max_price" placeholder="Max Price">
            <button type="submit">Search</button>
        </form>
        
        <h2>Results:</h2>
        {% for product in products %}
            <div class="product">
                <img src="{{ product.image }}" alt="{{ product.name }}">
                <h3><a href="{{ product.url }}" target="_blank">{{ product.name }}</a></h3>
                <p>Price: €{{ "%.2f"|format(product.price) }}</p>
                <p>Website: {{ product.website }}</p>
            </div>
        {% else %}
            <p>No products found.</p>
        {% endfor %}
    </body>
    </html>
    

    Running the Web Application

    Start the Flask app:

    python app.py
    

    Navigate to http://127.0.0.1:5000 in your browser to access the search interface.


    4. Automating the Scraping Process

    To keep the data up-to-date, schedule the scraping tasks periodically using cron (on Unix systems) or Task Scheduler (on Windows).

    Example Cron Job (Runs Daily at Midnight)

    1. Open the crontab:

      crontab -e
      
    2. Add the following line:

      0 0 * * * cd /path/to/product_scraper && scrapy crawl pccomponentes -a search_term=laptop && scrapy crawl coolmod -a search_term=laptop && scrapy crawl idealo -a search_term=laptop && scrapy crawl mediamarkt -a search_term=laptop
      

    Replace /path/to/product_scraper with your actual project directory and ensure all spiders accept necessary arguments.


    5. Best Practices and Considerations

    • Respect Robots.txt: Always check the website’s robots.txt file to understand what is allowed to be scraped.

    • Rate Limiting: Avoid sending too many requests in a short time. Scrapy handles this with settings like DOWNLOAD_DELAY.

      # settings.py
      DOWNLOAD_DELAY = 2  # 2 seconds delay
      
    • Error Handling: Implement robust error handling to manage unexpected issues during scraping.

    • Data Cleaning: Ensure that the data is cleaned and normalized before storing it in the database.

    • Legal Considerations: Ensure that scraping the target websites complies with their terms of service and local laws.

    • Scaling: For larger-scale scraping, consider using distributed systems like Scrapy Cluster or cloud-based solutions.


    Conclusion

    This guide provides a foundational framework for building a Python-based product search tool that scrapes multiple e-commerce websites and presents the data on a web interface. Depending on your specific needs and the complexity of the target websites, further customization and optimization may be necessary. Always remember to scrape responsibly and ethically.