Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-exp-03-25-grounding· GoogleHugging Face LMArena Search Arena 24K

what is LLMs.txt ?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-grounding’s answer.
  1. User

    what is LLMs.txt ?

  2. Gemini Assistant

    Based on the search results, "LLMs.txt" refers to a specific file format proposal, not just any text file related to Large Language Models (LLMs). Here's a breakdown:

    What LLMs.txt Is:

    • A Proposed Web Standard: "LLMs.txt" is a proposed standard for a file placed on websites specifically to provide information to Large Language Models (LLMs) in an easily digestible format. It was proposed by Jeremy Howard of Answer.AI in September 2024.
    • AI-Friendly Content: It's a markdown file (.md) designed to be both human-readable and easily parsed by AI systems. It aims to provide LLMs with concise background information, guidance, key content summaries, and structured links to more detailed information, bypassing complex HTML, JavaScript, or ads.
    • Optimization for LLMs: The purpose is to help LLMs understand and process website content more effectively, especially for large documentation sites, by overcoming challenges like limited context windows and noisy HTML data. Think of it as "SEO for AI".
    • Structure: The specification often includes two related files:
      • /llms.txt: Provides a simplified, structured overview of a site's navigation and key information (like site name and purpose).
      • /llms-full.txt: Compiles all or the most important documentation text into a single, comprehensive markdown file.
    • Location: Like robots.txt or sitemap.xml, the llms.txt file is intended to be placed in the root directory of a website (e.g., https://example.com/llms.txt).

    What LLMs.txt Does:

    • Streamlines Information: It distills a website's content into a concise, structured format optimized for AI processing.
    • Improves AI Understanding: Helps LLMs like ChatGPT, Perplexity, Claude, or Google Gemini to index and understand website content (especially documentation) more accurately and efficiently.
    • Facilitates AI Tasks: Enables AI tools to use website information more effectively, such as providing context-aware coding assistance or answering product-specific questions directly.
    • Complements Existing Standards: It serves a different purpose than robots.txt (which controls crawler access) or sitemap.xml (which lists pages for search engines). LLMs.txt focuses on providing curated content and context for LLM inference.

    How it's Used:

    • Currently, LLMs don't automatically discover these files like search engines crawl sitemaps. Users typically need to manually provide the file's content or URL to the LLM (e.g., via pasting, file upload).
    • Platforms like Mintlify automatically generate these files for documentation sites they host, significantly boosting the standard's adoption.
    • Tools and directories are emerging to help create and index sites with llms.txt files.

    In contrast, a generic text file (.txt) in the context of LLMs might be:

    • Training Data: Raw text files (.txt) can be used as part of the massive datasets LLMs are trained on, especially for unstructured text.
    • Input/Output: Simple text files can be used to provide input prompts to or capture output from an LLM.
    • Model Files (Less Common): While less common for the main model weights, associated configuration or metadata might sometimes be in text formats. More typical formats for model parameters include .bin, .pt, .gguf, .onnx.

    Therefore, LLMs.txt is a specific, emerging standard for structuring website information for AI consumption, distinct from general text files used in LLM training or operation.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.