What does browsercomp dataset do
- User
What does browsercomp dataset do
Assistant
BrowseComp is a benchmark created by OpenAI to measure the ability of AI agents to locate hard-to-find information by browsing the internet. Released on April 10, 2025, it stands for "Browsing Competition" and includes 1,266 challenging problems that require AI agents to search through many websites to find specific information[1][2].
The key characteristics of the BrowseComp dataset include:
- It focuses on questions where answers are short, specific, and there is (in principle) only a single correct answer, making evaluation straightforward[1]
- Questions are deliberately challenging - human trainers created questions that weren't solvable by another person within ten minutes[1][2]
- It tests an AI agent's ability to perform persistent web browsing, requiring creativity and strategic search approaches[1]
- Human trainers could only solve 29.2% of the problems in the dataset, with many taking 2-3 hours of research[1][2]
BrowseComp measures three critical skills for browsing agents:
- Reasoning about the factuality of internet content
- Persistence and depth of browsing capabilities
- Creative search strategies to find information efficiently[1]
When tested across different AI models, the results showed significant performance differences:
- GPT-4o: 0.6% accuracy
- GPT-4o with browsing: 1.9% accuracy
- GPT-4.5: 0.9% accuracy
- OpenAI o1: 9.9% accuracy
- Deep Research: 51.5% accuracy[1][2]
The benchmark is designed to be challenging yet easy to verify, serving as a useful metric for browsing agents similar to how programming competitions test coding agents. While it doesn't capture all aspects of real-world browsing tasks, it effectively measures a core capability - finding hard-to-locate information efficiently[1][2].
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.