Make an advanced and sophisticated python script - that's like a web-searcher but it…
- User
Make an advanced and sophisticated python script - that's like a web-searcher but it searches for horror games that doesn't have that much views, basically a hidden gem or underrated.
Without API's and such.
- Assistant
Creating an advanced and sophisticated Python script to search for underrated or hidden gem horror games without relying on APIs involves web scraping. Below is a comprehensive script that scrapes Steam's store page to find horror games with relatively low review counts, indicating they might be underrated.
Please Note:
- Respect Website’s Terms of Service: Before scraping any website, ensure that you're not violating its terms of service.
- Rate Limiting: Implement proper rate limiting to avoid overwhelming the server.
- Legal Considerations: Always ensure compliance with laws and regulations related to web scraping in your jurisdiction.
Prerequisites
- Python 3.x
- Required Libraries:
requestsfor making HTTP requests.BeautifulSoupfrombs4for parsing HTML content.pandasfor data manipulation and storage.tqdmfor displaying progress bars.
You can install the necessary libraries using pip:
pip install requests bs4 pandas tqdmScript Overview
-
Scrape Steam's Horror Genre Page:
- Steam organizes games by genres. We'll target the Horror genre.
- Extract game details like title, URL, number of reviews, and ratings.
-
Filter Underrated Games:
- Define criteria for "underrated," such as having fewer than a specified number of reviews or lower ratings.
-
Store Results:
- Save the filtered list of games to a CSV file for further analysis or use.
The Python Script
import requests from bs4 import BeautifulSoup import pandas as pd from tqdm import tqdm import time import random # Constants BASE_URL = "https://store.steampowered.com" GENRE_URL = f"{BASE_URL}/genre/Free%20to%20Play#p=0&tab=TopSellers" Horror_GENRE_URL = f"{BASE_URL}/genre/Free-to-Play//NoParent&genre=4060&snr=1_7_7_230_7" # Headers to mimic a browser visit HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" } def get_horror_games_page(page_number=1): """ Fetches the horror games page from Steam. Steam dynamically loads content, so we'll need to handle AJAX requests or find a static endpoint. However, for simplicity, we'll use the search results page that can be paginated via URL parameters. """ # Construct URL with pagination url = f"{BASE_URL}/search/results/?genres=4020,4060&category1=998&sort_by=Released_DESC&page={page_number}" response = requests.get(url, headers=HEADERS) if response.status_code != 200: print(f"Failed to retrieve page {page_number}") return None return response.text def parse_games(html_content): """ Parses the HTML content to extract game details. """ soup = BeautifulSoup(html_content, 'html.parser') game_elements = soup.find_all('a', {'class': 'search_result_row'}) games = [] for game in game_elements: try: title = game.find('span', {'class': 'title'}).text.strip() url = game['href'] release_date = game.find('div', {'class': 'search_released'}).text.strip() # Ratings review_summary = game.find('span', {'class': 'search_review_summary'}) if review_summary and 'data-tooltip-html' in review_summary.attrs: tooltip = review_summary['data-tooltip-html'] # Example tooltip: "Very Positive<br>10,123 reviews" tooltip_parts = tooltip.split('<br>') if len(tooltip_parts) == 2: sentiment = tooltip_parts[0] review_count = tooltip_parts[1].split(' ')[0].replace(',', '') else: sentiment = "No Reviews" review_count = "0" else: sentiment = "No Reviews" review_count = "0" review_count = int(review_count) # Price price_div = game.find('div', {'class': 'search_price'}) price_text = price_div.text.strip() if price_text == "": price = "Free" else: price = price_text.split('₽')[0].strip() # Adjust currency if needed games.append({ 'Title': title, 'URL': url, 'Release Date': release_date, 'Review Sentiment': sentiment, 'Review Count': review_count, 'Price': price }) except Exception as e: print(f"Error parsing a game: {e}") return games def is_underrated(game, review_threshold=100): """ Determines if a game is underrated based on review count. """ return game['Review Count'] < review_threshold def main(): all_games = [] total_pages = 5 # Adjust based on how many pages you want to scrape print("Starting to scrape Steam Horror Games...") for page in tqdm(range(1, total_pages + 1)): html = get_horror_games_page(page) if not html: continue games = parse_games(html) all_games.extend(games) # Be polite with the server time.sleep(random.uniform(1, 3)) print(f"Total games scraped: {len(all_games)}") # Convert to DataFrame df = pd.DataFrame(all_games) # Filter for underrated games underrated_threshold = 100 # Define what "underrated" means underrated_games = df[df['Review Count'] < underrated_threshold] print(f"Underrated games found: {len(underrated_games)}") # Save to CSV underrated_games.to_csv('underrated_horror_games.csv', index=False) print("Saved underrated horror games to 'underrated_horror_games.csv'") if __name__ == "__main__": main()Script Explanation
-
Libraries Imported:
requeststo handle HTTP requests.BeautifulSoupto parse and navigate HTML content.pandasto structure and manipulate data.tqdmto display a progress bar during scraping.timeandrandomto implement delays between requests, mimicking human browsing and preventing server overload.
-
Constants Defined:
BASE_URLis the base Steam store URL.GENRE_URLandHorror_GENRE_URLare meant to target specific genre pages. Adjust these based on the actual Steam URL structure.HEADERSmimic a legitimate browser to prevent blocking.
-
Functions:
get_horror_games_page: Constructs the URL for a specific page of horror games and fetches its HTML content.parse_games: Parses the HTML to extract relevant game information, such as title, URL, release date, review sentiment, review count, and price.is_underrated: A helper function to determine if a game is underrated based on a review count threshold.
-
Main Execution Flow:
- Scrapes a predefined number of pages (
total_pages). - Parses each page to extract game data.
- Aggregates all games into a DataFrame.
- Filters the DataFrame to include only games that meet the underrated criteria.
- Saves the result to a CSV file.
- Scrapes a predefined number of pages (
Customization and Enhancements
- Dynamic Pagination: You can enhance the script to dynamically determine the number of pages by inspecting the total number of results.
- Advanced Filtering: Incorporate additional filters such as ratings (e.g., "Very Positive"), price ranges, or release dates to refine your hidden gems.
- User Input: Allow users to input criteria like the review threshold or number of pages to scrape.
- Error Handling: Improve error handling to retry failed requests or skip problematic entries gracefully.
- Multiprocessing: For faster scraping, consider using multiprocessing or asynchronous requests (
aiohttpandasyncio). However, ensure you're not violating Steam's usage policies.
Caveats and Considerations
- Dynamic Content: Steam's website may use JavaScript to load some content dynamically. If certain data isn't appearing, consider using tools like Selenium or Playwright to render JavaScript. However, these come with added complexity and performance overhead.
- Change in Website Structure: Websites frequently update their HTML structures, which can break your scraper. Regularly maintain and update your scraping logic accordingly.
- Legal and Ethical Scraping: Always ensure that your scraping activities are legal and ethical. Excessive scraping can lead to IP bans or legal actions.
Conclusion
The provided script offers a robust starting point for scraping Steam's store to find underrated horror games. By adjusting parameters and enhancing functionality, you can tailor it to better suit your specific needs. Always remember to scrape responsibly and respect the target website's policies.