Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Make an advanced and sophisticated python script - that's like a web-searcher but it…

  1. User

    Make an advanced and sophisticated python script - that's like a web-searcher but it searches for horror games that doesn't have that much views, basically a hidden gem or underrated.

    Without API's and such.

  2. Assistant

    Creating an advanced and sophisticated Python script to search for underrated or hidden gem horror games without relying on APIs involves web scraping. Below is a comprehensive script that scrapes Steam's store page to find horror games with relatively low review counts, indicating they might be underrated.

    Please Note:

    • Respect Website’s Terms of Service: Before scraping any website, ensure that you're not violating its terms of service.
    • Rate Limiting: Implement proper rate limiting to avoid overwhelming the server.
    • Legal Considerations: Always ensure compliance with laws and regulations related to web scraping in your jurisdiction.

    Prerequisites

    1. Python 3.x
    2. Required Libraries:
      • requests for making HTTP requests.
      • BeautifulSoup from bs4 for parsing HTML content.
      • pandas for data manipulation and storage.
      • tqdm for displaying progress bars.

    You can install the necessary libraries using pip:

    pip install requests bs4 pandas tqdm
    

    Script Overview

    1. Scrape Steam's Horror Genre Page:

      • Steam organizes games by genres. We'll target the Horror genre.
      • Extract game details like title, URL, number of reviews, and ratings.
    2. Filter Underrated Games:

      • Define criteria for "underrated," such as having fewer than a specified number of reviews or lower ratings.
    3. Store Results:

      • Save the filtered list of games to a CSV file for further analysis or use.

    The Python Script

    import requests
    from bs4 import BeautifulSoup
    import pandas as pd
    from tqdm import tqdm
    import time
    import random
    
    # Constants
    BASE_URL = "https://store.steampowered.com"
    GENRE_URL = f"{BASE_URL}/genre/Free%20to%20Play#p=0&tab=TopSellers"
    Horror_GENRE_URL = f"{BASE_URL}/genre/Free-to-Play//NoParent&genre=4060&snr=1_7_7_230_7"
    
    # Headers to mimic a browser visit
    HEADERS = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
    }
    
    def get_horror_games_page(page_number=1):
        """
        Fetches the horror games page from Steam.
        Steam dynamically loads content, so we'll need to handle AJAX requests or find a static endpoint.
        However, for simplicity, we'll use the search results page that can be paginated via URL parameters.
        """
        # Construct URL with pagination
        url = f"{BASE_URL}/search/results/?genres=4020,4060&category1=998&sort_by=Released_DESC&page={page_number}"
        response = requests.get(url, headers=HEADERS)
        if response.status_code != 200:
            print(f"Failed to retrieve page {page_number}")
            return None
        return response.text
    
    def parse_games(html_content):
        """
        Parses the HTML content to extract game details.
        """
        soup = BeautifulSoup(html_content, 'html.parser')
        game_elements = soup.find_all('a', {'class': 'search_result_row'})
        games = []
        for game in game_elements:
            try:
                title = game.find('span', {'class': 'title'}).text.strip()
                url = game['href']
                release_date = game.find('div', {'class': 'search_released'}).text.strip()
                # Ratings
                review_summary = game.find('span', {'class': 'search_review_summary'})
                if review_summary and 'data-tooltip-html' in review_summary.attrs:
                    tooltip = review_summary['data-tooltip-html']
                    # Example tooltip: "Very Positive<br>10,123 reviews"
                    tooltip_parts = tooltip.split('<br>')
                    if len(tooltip_parts) == 2:
                        sentiment = tooltip_parts[0]
                        review_count = tooltip_parts[1].split(' ')[0].replace(',', '')
                    else:
                        sentiment = "No Reviews"
                        review_count = "0"
                else:
                    sentiment = "No Reviews"
                    review_count = "0"
                review_count = int(review_count)
                
                # Price
                price_div = game.find('div', {'class': 'search_price'})
                price_text = price_div.text.strip()
                if price_text == "":
                    price = "Free"
                else:
                    price = price_text.split('₽')[0].strip()  # Adjust currency if needed
    
                games.append({
                    'Title': title,
                    'URL': url,
                    'Release Date': release_date,
                    'Review Sentiment': sentiment,
                    'Review Count': review_count,
                    'Price': price
                })
            except Exception as e:
                print(f"Error parsing a game: {e}")
        return games
    
    def is_underrated(game, review_threshold=100):
        """
        Determines if a game is underrated based on review count.
        """
        return game['Review Count'] < review_threshold
    
    def main():
        all_games = []
        total_pages = 5  # Adjust based on how many pages you want to scrape
        print("Starting to scrape Steam Horror Games...")
        for page in tqdm(range(1, total_pages + 1)):
            html = get_horror_games_page(page)
            if not html:
                continue
            games = parse_games(html)
            all_games.extend(games)
            # Be polite with the server
            time.sleep(random.uniform(1, 3))
        
        print(f"Total games scraped: {len(all_games)}")
    
        # Convert to DataFrame
        df = pd.DataFrame(all_games)
        
        # Filter for underrated games
        underrated_threshold = 100  # Define what "underrated" means
        underrated_games = df[df['Review Count'] < underrated_threshold]
    
        print(f"Underrated games found: {len(underrated_games)}")
    
        # Save to CSV
        underrated_games.to_csv('underrated_horror_games.csv', index=False)
        print("Saved underrated horror games to 'underrated_horror_games.csv'")
    
    if __name__ == "__main__":
        main()
    

    Script Explanation

    1. Libraries Imported:

      • requests to handle HTTP requests.
      • BeautifulSoup to parse and navigate HTML content.
      • pandas to structure and manipulate data.
      • tqdm to display a progress bar during scraping.
      • time and random to implement delays between requests, mimicking human browsing and preventing server overload.
    2. Constants Defined:

      • BASE_URL is the base Steam store URL.
      • GENRE_URL and Horror_GENRE_URL are meant to target specific genre pages. Adjust these based on the actual Steam URL structure.
      • HEADERS mimic a legitimate browser to prevent blocking.
    3. Functions:

      • get_horror_games_page: Constructs the URL for a specific page of horror games and fetches its HTML content.
      • parse_games: Parses the HTML to extract relevant game information, such as title, URL, release date, review sentiment, review count, and price.
      • is_underrated: A helper function to determine if a game is underrated based on a review count threshold.
    4. Main Execution Flow:

      • Scrapes a predefined number of pages (total_pages).
      • Parses each page to extract game data.
      • Aggregates all games into a DataFrame.
      • Filters the DataFrame to include only games that meet the underrated criteria.
      • Saves the result to a CSV file.

    Customization and Enhancements

    • Dynamic Pagination: You can enhance the script to dynamically determine the number of pages by inspecting the total number of results.
    • Advanced Filtering: Incorporate additional filters such as ratings (e.g., "Very Positive"), price ranges, or release dates to refine your hidden gems.
    • User Input: Allow users to input criteria like the review threshold or number of pages to scrape.
    • Error Handling: Improve error handling to retry failed requests or skip problematic entries gracefully.
    • Multiprocessing: For faster scraping, consider using multiprocessing or asynchronous requests (aiohttp and asyncio). However, ensure you're not violating Steam's usage policies.

    Caveats and Considerations

    • Dynamic Content: Steam's website may use JavaScript to load some content dynamically. If certain data isn't appearing, consider using tools like Selenium or Playwright to render JavaScript. However, these come with added complexity and performance overhead.
    • Change in Website Structure: Websites frequently update their HTML structures, which can break your scraper. Regularly maintain and update your scraping logic accordingly.
    • Legal and Ethical Scraping: Always ensure that your scraping activities are legal and ethical. Excessive scraping can lead to IP bans or legal actions.

    Conclusion

    The provided script offers a robust starting point for scraping Steam's store to find underrated horror games. By adjusting parameters and enhancing functionality, you can tailor it to better suit your specific needs. Always remember to scrape responsibly and respect the target website's policies.