I need a fairly efficient Python program used to search products on certain websites,…
- User
I need a fairly efficient Python program used to search products on certain websites, primarily laptops, computer components and phones. The end result would be a basic website where the user can search for a product name, then filter the results. The results come from performing searches on the best technology related websites from Spain, which are PcComponentes, Coolmod, Idealo and MediaMarkt. The program should make use of already existing scraping systems in order to find data like the search bar, product names, prices, filters, etc, on an organized and easy manner, and it should try to stay, at least, a bit undetected in order to extract this small amount of data for the user.
- Assistant
Creating a Python program to scrape product data from multiple Spanish e-commerce websites (PcComponentes, Coolmod, Idealo, MediaMarkt) and display it on a basic web interface involves several steps:
- Web Scraping: Extracting data from the target websites.
- Data Storage: Storing the scraped data in a database.
- Web Interface: Building a user-friendly website to search and filter products.
Below is a comprehensive guide to building this system using Python, leveraging existing libraries and frameworks to ensure efficiency and maintainability.
1. Web Scraping
Tools and Libraries
- Scrapy: A powerful and efficient web scraping framework.
- BeautifulSoup: For parsing HTML documents (alternative to Scrapy selectors).
- Requests: To make HTTP requests.
- Selenium: For websites that heavily rely on JavaScript (optional).
- Scrapy-UserAgents and Scrapy-Proxies: To rotate user agents and proxies for staying undetected.
Setup
First, install the necessary libraries:
pip install scrapy scrapy-user-agents scrapy-proxies beautifulsoup4 requests seleniumScrapy Project Structure
Create a Scrapy project:
scrapy startproject product_scraper cd product_scraperExample Spider for PcComponentes
Each website will require a separate spider due to differing structures. Below is an example spider for PcComponentes.
# product_scraper/spiders/pccomponentes_spider.py import scrapy from scrapy import Request from urllib.parse import urlencode class PcComponentesSpider(scrapy.Spider): name = "pccomponentes" allowed_domains = ["www.pccomponentes.com"] def __init__(self, search_term='', *args, **kwargs): super(PcComponentesSpider, self).__init__(*args, **kwargs) self.search_term = search_term self.start_urls = [f"https://www.pccomponentes.com/busqueda/{self.search_term}"] def parse(self, response): products = response.css('div.product-wrapper') for product in products: yield { 'name': product.css('h2.product-name a::text').get().strip(), 'price': product.css('span.price::text').get().strip(), 'url': product.css('h2.product-name a::attr(href)').get(), 'image': product.css('img::attr(src)').get() } # Pagination next_page = response.css('a.next::attr(href)').get() if next_page: yield Request(url=next_page, callback=self.parse)Handling Other Websites
Similarly, create spiders for Coolmod, Idealo, and MediaMarkt by analyzing their HTML structure and adjusting the selectors accordingly.
Middleware for Staying Undetected
Edit
settings.pyto include middleware for rotating user agents and proxies.# product_scraper/settings.py DOWNLOADER_MIDDLEWARES = { 'scrapy_user_agents.middleware.RandomUserAgentMiddleware': 400, 'scrapy_proxies.RandomProxy': 100, 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110, } # User Agents USER_AGENTS = [ # List of user agents ] # Proxies PROXY_LIST = '/path/to/proxy/list.txt' PROXY_MODE = 0Ensure you have a list of proxies in
proxy/list.txt. Services like ScraperAPI or ProxyScrape can provide proxies.Running the Spider
You can run the spider with a search term like this:
scrapy crawl pccomponentes -a search_term=laptop -o data/pccomponentes.jsonAutomating Scraping for All Websites
Create separate spiders and run them sequentially or concurrently using a master script or Scrapy’s
CrawlSpider.
2. Data Storage
Use SQLite for simplicity or PostgreSQL for scalability. Below is an example using SQLite with SQLAlchemy.
Install SQLAlchemy
pip install sqlalchemyDefine Models
# product_scraper/models.py from sqlalchemy import create_engine, Column, String, Float, Integer from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker Base = declarative_base() class Product(Base): __tablename__ = 'products' id = Column(Integer, primary_key=True) website = Column(String) name = Column(String) price = Column(Float) url = Column(String) image = Column(String) # Database setup engine = create_engine('sqlite:///products.db') Base.metadata.create_all(engine) Session = sessionmaker(bind=engine)Storing Scraped Data
Modify the spiders to store data into the database instead of outputting to JSON.
# product_scraper/spiders/pccomponentes_spider.py import scrapy from product_scraper.models import Product, Session class PcComponentesSpider(scrapy.Spider): name = "pccomponentes" allowed_domains = ["www.pccomponentes.com"] def __init__(self, search_term='', *args, **kwargs): super(PcComponentesSpider, self).__init__(*args, **kwargs) self.search_term = search_term self.start_urls = [f"https://www.pccomponentes.com/busqueda/{self.search_term}"] self.session = Session() def parse(self, response): products = response.css('div.product-wrapper') for product in products: name = product.css('h2.product-name a::text').get().strip() price_text = product.css('span.price::text').get().strip() price = float(price_text.replace('€', '').replace(',', '.')) url = product.css('h2.product-name a::attr(href)').get() image = product.css('img::attr(src)').get() new_product = Product( website='PcComponentes', name=name, price=price, url=url, image=image ) self.session.add(new_product) self.session.commit() # Pagination next_page = response.css('a.next::attr(href)').get() if next_page: yield response.follow(next_page, callback=self.parse) def closed(self, reason): self.session.close()Repeat similar modifications for other spiders, ensuring the
websitefield is appropriately set.
3. Building the Web Interface
Use Flask to create a simple web application where users can search and filter products.
Install Flask and SQLAlchemy
pip install flask sqlalchemyCreate the Flask App
# app.py from flask import Flask, render_template, request from sqlalchemy import create_engine, or_ from sqlalchemy.orm import sessionmaker from product_scraper.models import Product, Base app = Flask(__name__) # Database setup engine = create_engine('sqlite:///products.db') Base.metadata.bind = engine DBSession = sessionmaker(bind=engine) session = DBSession() @app.route('/', methods=['GET', 'POST']) def index(): if request.method == 'POST': search_term = request.form.get('search') filter_min = request.form.get('min_price', type=float) filter_max = request.form.get('max_price', type=float) query = session.query(Product).filter(Product.name.ilike(f'%{search_term}%')) if filter_min is not None: query = query.filter(Product.price >= filter_min) if filter_max is not None: query = query.filter(Product.price <= filter_max) results = query.all() return render_template('index.html', products=results, search=search_term) return render_template('index.html', products=[], search='') if __name__ == '__main__': app.run(debug=True)Create the HTML Template
Create a
templatesfolder in the same directory asapp.pyand addindex.html:<!-- templates/index.html --> <!DOCTYPE html> <html> <head> <title>Product Search</title> <style> /* Basic styling */ body { font-family: Arial, sans-serif; } .product { border: 1px solid #ddd; padding: 10px; margin: 10px; } .product img { max-width: 150px; } </style> </head> <body> <h1>Product Search</h1> <form method="POST"> <input type="text" name="search" placeholder="Product name" value="{{ search }}" required> <input type="number" step="0.01" name="min_price" placeholder="Min Price"> <input type="number" step="0.01" name="max_price" placeholder="Max Price"> <button type="submit">Search</button> </form> <h2>Results:</h2> {% for product in products %} <div class="product"> <img src="{{ product.image }}" alt="{{ product.name }}"> <h3><a href="{{ product.url }}" target="_blank">{{ product.name }}</a></h3> <p>Price: €{{ "%.2f"|format(product.price) }}</p> <p>Website: {{ product.website }}</p> </div> {% else %} <p>No products found.</p> {% endfor %} </body> </html>Running the Web Application
Start the Flask app:
python app.pyNavigate to
http://127.0.0.1:5000in your browser to access the search interface.
4. Automating the Scraping Process
To keep the data up-to-date, schedule the scraping tasks periodically using cron (on Unix systems) or Task Scheduler (on Windows).
Example Cron Job (Runs Daily at Midnight)
-
Open the crontab:
crontab -e -
Add the following line:
0 0 * * * cd /path/to/product_scraper && scrapy crawl pccomponentes -a search_term=laptop && scrapy crawl coolmod -a search_term=laptop && scrapy crawl idealo -a search_term=laptop && scrapy crawl mediamarkt -a search_term=laptop
Replace
/path/to/product_scraperwith your actual project directory and ensure all spiders accept necessary arguments.
5. Best Practices and Considerations
-
Respect Robots.txt: Always check the website’s
robots.txtfile to understand what is allowed to be scraped. -
Rate Limiting: Avoid sending too many requests in a short time. Scrapy handles this with settings like
DOWNLOAD_DELAY.# settings.py DOWNLOAD_DELAY = 2 # 2 seconds delay -
Error Handling: Implement robust error handling to manage unexpected issues during scraping.
-
Data Cleaning: Ensure that the data is cleaned and normalized before storing it in the database.
-
Legal Considerations: Ensure that scraping the target websites complies with their terms of service and local laws.
-
Scaling: For larger-scale scraping, consider using distributed systems like Scrapy Cluster or cloud-based solutions.
Conclusion
This guide provides a foundational framework for building a Python-based product search tool that scrapes multiple e-commerce websites and presents the data on a web interface. Depending on your specific needs and the complexity of the target websites, further customization and optimization may be necessary. Always remember to scrape responsibly and ethically.