Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

I'd like to let my site being crawled, but not aggressively. Lots of bots do not respect…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o4-mini-2025-04-16’s answer.
  1. User

    I'd like to let my site being crawled, but not aggressively. Lots of bots do not respect the robots.txt nor send a proper user agent. What would you do to limit them in nginx? (and do you see other ways outside nginx?)

  2. Assistant

    Below is a “menu” of practical things you can do.
    Feel free to mix‑and‑match; most people end up using 2 – 3 of them.

    ────────────────────

    1. Throttle in Nginx ──────────────────── A. Global IP‑based rate‑limit
      (works even when the UA string is blank or forged).
    # 60 requests per minute, per IP
    limit_req_zone $binary_remote_addr zone=perip:20m rate=60r/m;
    
    server {
        ...
    
        # Apply to every URI
        location / {
            limit_req zone=perip burst=30 nodelay;   # burst lets short spikes through
        }
    }
    

    B. Give reputable crawlers more room
    (so Google/Bing aren’t slowed, but random scrapers are).

    # 1. Categorise UA
    map $http_user_agent $crawler_tier {
        default                          slow;    # everything we don't explicitly whitelist
        "~*(Googlebot|Bingbot|DuckDuckBot|Slurp)" fast;
    }
    
    # 2. Choose the key we’ll limit on:  
    #    real crawlers → empty key → not rate‑limited
    #    others        → IP address  → rate‑limited
    map $crawler_tier $rl_key {
        fast "";                   # empty = no limiting
        slow $binary_remote_addr;  # key = IP
    }
    
    # 3. Declare the zone
    limit_req_zone $rl_key zone=crawlers:20m rate=30r/m;  # 0.5 r/s
    
    server {
        location / {
            limit_req zone=crawlers burst=60 nodelay;
        }
    }
    

    C. Connection limiter (helps if a bot opens hundreds of keep‑alive sockets):

    limit_conn_zone $binary_remote_addr zone=connip:20m;
    
    server {
        limit_conn connip 10;   # max 10 simultaneous connections per IP
    }
    

    Tips
    • Return 429 or 503 + Retry-After so well‑behaved clients know to back off.
    • Log to a dedicated file (access_log bot.log if=$is_bot;) so you can tune numbers later.

    ──────────────────── 2. “Bad UA” or “empty UA” blocking ────────────────────

    map $http_user_agent $block_ua {
        default                          0;
        ""                               1;                         # empty UA
        "~*(MJ12bot|AhrefsBot|BLEXBot)"  1;                         # serial offenders
    }
    
    server {
        if ($block_ua) { return 403; }   # or 429 if you prefer
    }
    

    (Be careful with long regex lists; use map once, not if per bot.)

    ──────────────────── 3. Outside of Nginx ──────────────────── A. fail2ban
    • Tail the access log, look for “429 too many requests”, ban the IP in iptables for e.g. 1 h.
    • Good for bots that just ignore your 429.

    B. iptables/nftables “recent” or “hashlimit”
    – Kernel‑level rate‑limit before the request even reaches Nginx.
    – Example (nft):
    ct count over 200 per 30s drop

    C. WAF / CDN / Reverse‑proxy (Cloudflare, Fastly, Akamai, etc.)
    • Built‑in bot management & automatic challenge pages.
    • Very little maintenance on your side, but you pay for it.

    D. ModSecurity in “anomaly scoring” mode
    • Score headers, missing Accept‑Language, weird crawl patterns, etc., and block when score>×.

    E. CAPTCHA or token gating on the expensive endpoints
    • Not ideal for public pages, but perfect for search forms, API endpoints, etc.

    F. Honeypot link + auto‑ban
    • Add an invisible link (display:none).
    • Any agent that fetches it obviously ignores robots & CSS → add to blocklist.

    ──────────────────── 4. Monitoring & tuning ────────────────────

    1. log_format crawl '$remote_addr $http_user_agent $request_time';
      Pipe through Grafana/Prometheus or even a simple awk + gnuplot.
    2. Start conservative (e.g. 120 r/m) and tighten once you know “normal” traffic.
    3. Check Google Search Console → Crawl stats → make sure Google isn’t getting 429s.

    ──────────────────── 5. Quick copy‑paste starter ────────────────────

    # ==== /etc/nginx/conf.d/ratelimit.conf ==============================
    # 30 requests per minute unless the UA matches our "good" list
    map $http_user_agent $rl_key {
        "~*(Googlebot|Bingbot|DuckDuckBot|Slurp)" "";
        default                                    $binary_remote_addr;
    }
    
    limit_req_zone $rl_key zone=crawlers:20m rate=30r/m;
    limit_conn_zone $binary_remote_addr zone=connip:20m;
    
    server {
        ...
        location / {
            limit_conn connip 15;
            limit_req  zone=crawlers burst=60 nodelay;
        }
    
        # Empty UA / known bad bots → 403
        map $http_user_agent $badua {
            default 0;
            "" 1;
            "~*(AhrefsBot|MJ12bot|BLEXBot)" 1;
        }
        if ($badua) { return 403; }
    }
    # ====================================================================
    

    That setup alone stops 90 % of “drive‑by” scrapers without hurting the good guys.
    Add fail2ban or a CDN in front if it still feels heavy. Happy crawling‑but‑not‑too‑much! 🕷️🛑

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.