Skip to content
TheChatArchiveThe Chat Archive
chatgpt-4o-latest-20250326· OpenAIHugging Face LMArena Human Preference 140K

Is there a good way to scrape LMArena's site?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-opus-4-20250514’s answer.
  1. User

    Is there a good way to scrape LMArena's site?

  2. Assistant

    Scraping LMArena (or any website) depends on the site's structure, terms of service, and whether it uses protections like JavaScript rendering or anti-bot mechanisms. Before going into how to scrape it, let's consider a few key points.


    ⚖️ Legal Disclaimer

    Before scraping any website:

    1. Check the site's robots.txt file (e.g., lmarena.com/robots.txt) to see if scraping is allowed.
    2. Read the site's Terms of Service to ensure you're not violating any rules.
    3. Use scraping responsibly (rate limiting, avoiding impact on servers).

    If LMArena restricts scraping or access through automated means, you should respect that.


    🔧 Tools You Might Use

    1. BeautifulSoup + Requests (for simple static pages)

    If the site content is static HTML, a simple scraper might look like this:

    import requests
    from bs4 import BeautifulSoup
    
    url = "https://www.lmarena.com/fixtures"
    headers = {
        "User-Agent": "Mozilla/5.0"
    }
    
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, "html.parser")
    
    # Example: extracting match fixtures
    matches = soup.find_all("div", class_="match-container")
    for match in matches:
        team1 = match.find("div", class_="team1").text.strip()
        team2 = match.find("div", class_="team2").text.strip()
        time = match.find("span", class_="match-time").text.strip()
        print(f"{team1} vs {team2} at {time}")
    

    But this depends entirely on the HTML structure of the actual site.


    2. Selenium or Playwright (for JavaScript-rendered pages)

    If LMArena loads content dynamically (using JS), you’ll need a headless browser:

    Selenium Example:

    from selenium import webdriver
    from bs4 import BeautifulSoup
    import time
    
    driver = webdriver.Chrome()  # Or use undetected_chromedriver for stealth
    driver.get("https://www.lmarena.com/fixtures")
    
    time.sleep(5)  # Wait for JS content to load
    
    soup = BeautifulSoup(driver.page_source, "html.parser")
    
    # Find relevant data
    fixtures = soup.select(".fixture-container")
    for fixture in fixtures:
        print(fixture.text)
    
    driver.quit()
    

    3. Playwright (modern alternative to Selenium)

    Supports better JavaScript handling and works asynchronously.

    from playwright.sync_api import sync_playwright
    
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto("https://www.lmarena.com/fixtures")
    
        # Wait for necessary content
        page.wait_for_selector(".fixture-container")
    
        html = page.content()
        soup = BeautifulSoup(html, "html.parser")
    
        fixtures = soup.select(".fixture-container")
        for fixture in fixtures:
            print(fixture.text)
    
        browser.close()
    

    🔄 Alternative: Look for an API

    LMArena might be loading the data via AJAX. You can inspect this using browser developer tools:

    1. Go to Network tab > XHR or Fetch filter.
    2. Refresh the page — see if the match data is being loaded from a .json or .php? request.
    3. You can spoof requests to those endpoints instead — faster & cleaner.

    ❌ Challenges You Might Face

    • CAPTCHA or Cloudflare protection
    • Obfuscated JavaScript
    • Rate limiting or IP blacklisting

    Solution: Use proxies, undetected-chromedriver, or stealth plugins for Selenium/Playwright.


    ✅ Best Practice

    If you need regular data, consider:


    Want Help With a Specific Page?

    If you link me to a specific LMArena page or describe what data you want (e.g., fixtures, scores, streams), I can give you tailored code.


    Would you like me to walk you through using Playwright or checking the network requests from dev tools?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.