Skip to content
TheChatArchiveThe Chat Archive
o4-mini-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

I'd like to let my site being crawled, but not aggressively. Lots of bots do not respect…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o3-2025-04-16’s answer.
  1. User

    I'd like to let my site being crawled, but not aggressively. Lots of bots do not respect the robots.txt nor send a proper user agent. What would you do to limit them in nginx? (and do you see other ways outside nginx?)

  2. Assistant

    Here’s a two‑pronged approach:

    1. throttle/block “bad” crawlers at the Nginx edge
    2. add extra layers (CDN/WAF, honeypots, behavioural checks, etc.)

    1) NGINX‑Level Controls

    a) Rate‑limit per IP

    Prevent any single IP from hammering your site:

    http {
      # share a small in‑memory store (10 MB) and cap at ~1 req/sec
      limit_req_zone $binary_remote_addr zone=bot_limit:10m rate=1r/s;
    
      server {
        ...
        location / {
          # allow small burst of 5, then start delaying/limiting
          limit_req zone=bot_limit burst=5 nodelay;
        }
      }
    }
    

    → Good for “slow down aggressive scrapers”

    b) Limit concurrent connections

    http {
      limit_conn_zone $binary_remote_addr zone=addr:10m;
    
      server {
        ...
        location / {
          # no more than 10 simultaneous connections per IP
          limit_conn addr 10;
        }
      }
    }
    

    c) Block “no‑UA” & known bad agents

    server {
      ...
      # block requests missing a UA
      if ($http_user_agent = "") {
        return 403;
      }
    
      # block specific crawler signatures
      if ($http_user_agent ~* (masscan|python-requests|libwww-perl)) {
        return 444;
      }
    }
    

    d) Honeypot “trap” URLs

    Insert hidden links in your HTML/CSS that real browsers never load; bots that request them get black‑holed.

    <!-- in your <head> -->
    <link rel="stylesheet" href="/css/styles.css">
    <div style="display:none">
      <img src="/bot_trap/does-not-exist" alt="">
    </div>
    
    location /bot_trap/ {
      return 444;  # kills the connection
    }
    

    2) Beyond NGINX

    1. CDN / Bot‑Management services
      • Cloudflare Bot Management / AWS WAF / Akamai – can fingerprint & challenge headless scrapers.
    2. CAPTCHA / JavaScript challenges
      • Show a JS or CAPTCHA step after X requests/minute.
    3. Fail2ban + logs
      • Scan Nginx logs for 429/444/403 events, auto‑block abusive IPs via iptables.
    4. API Gateway / Reverse Proxy
      • Use Kong, Tyk, etc. for fine‑grained rate‑limits & quotas per API key.
    5. Behavioral analytics
      • Monitor mouse/scroll/timing patterns; flag unrealistic “read” speeds.
    6. Authentication / API Keys
      • Only allow authenticated sessions or tokenized requests for data‑heavy endpoints.

    👣 Next steps:
    • Tune rate & burst to fit your traffic profile.
    • Monitor limit_req_status & limit_conn_status in your access logs.
    • Roll out in staging first to avoid false‑positives against real users.

    Let me know if you want detailed examples for any of the above!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.