I'd like to let my site being crawled, but not aggressively. Lots of bots do not respect…
- User
I'd like to let my site being crawled, but not aggressively. Lots of bots do not respect the robots.txt nor send a proper user agent. What would you do to limit them in nginx? (and do you see other ways outside nginx?)
- Assistant
Below is a “menu” of practical things you can do.
Feel free to mix‑and‑match; most people end up using 2 – 3 of them.────────────────────
- Throttle in Nginx
────────────────────
A. Global IP‑based rate‑limit
(works even when the UA string is blank or forged).
# 60 requests per minute, per IP limit_req_zone $binary_remote_addr zone=perip:20m rate=60r/m; server { ... # Apply to every URI location / { limit_req zone=perip burst=30 nodelay; # burst lets short spikes through } }B. Give reputable crawlers more room
(so Google/Bing aren’t slowed, but random scrapers are).# 1. Categorise UA map $http_user_agent $crawler_tier { default slow; # everything we don't explicitly whitelist "~*(Googlebot|Bingbot|DuckDuckBot|Slurp)" fast; } # 2. Choose the key we’ll limit on: # real crawlers → empty key → not rate‑limited # others → IP address → rate‑limited map $crawler_tier $rl_key { fast ""; # empty = no limiting slow $binary_remote_addr; # key = IP } # 3. Declare the zone limit_req_zone $rl_key zone=crawlers:20m rate=30r/m; # 0.5 r/s server { location / { limit_req zone=crawlers burst=60 nodelay; } }C. Connection limiter (helps if a bot opens hundreds of keep‑alive sockets):
limit_conn_zone $binary_remote_addr zone=connip:20m; server { limit_conn connip 10; # max 10 simultaneous connections per IP }Tips
• Return 429 or 503 +Retry-Afterso well‑behaved clients know to back off.
• Log to a dedicated file (access_log bot.log if=$is_bot;) so you can tune numbers later.──────────────────── 2. “Bad UA” or “empty UA” blocking ────────────────────
map $http_user_agent $block_ua { default 0; "" 1; # empty UA "~*(MJ12bot|AhrefsBot|BLEXBot)" 1; # serial offenders } server { if ($block_ua) { return 403; } # or 429 if you prefer }(Be careful with long regex lists; use
maponce, notifper bot.)──────────────────── 3. Outside of Nginx ──────────────────── A. fail2ban
• Tail the access log, look for “429 too many requests”, ban the IP iniptablesfor e.g. 1 h.
• Good for bots that just ignore your 429.B. iptables/nftables “recent” or “hashlimit”
– Kernel‑level rate‑limit before the request even reaches Nginx.
– Example (nft):
ct count over 200 per 30s dropC. WAF / CDN / Reverse‑proxy (Cloudflare, Fastly, Akamai, etc.)
• Built‑in bot management & automatic challenge pages.
• Very little maintenance on your side, but you pay for it.D. ModSecurity in “anomaly scoring” mode
• Score headers, missing Accept‑Language, weird crawl patterns, etc., and block when score>×.E. CAPTCHA or token gating on the expensive endpoints
• Not ideal for public pages, but perfect for search forms, API endpoints, etc.F. Honeypot link + auto‑ban
• Add an invisible link (display:none).
• Any agent that fetches it obviously ignores robots & CSS → add to blocklist.──────────────────── 4. Monitoring & tuning ────────────────────
log_format crawl '$remote_addr $http_user_agent $request_time';
Pipe through Grafana/Prometheus or even a simpleawk+gnuplot.- Start conservative (e.g. 120 r/m) and tighten once you know “normal” traffic.
- Check Google Search Console → Crawl stats → make sure Google isn’t getting 429s.
──────────────────── 5. Quick copy‑paste starter ────────────────────
# ==== /etc/nginx/conf.d/ratelimit.conf ============================== # 30 requests per minute unless the UA matches our "good" list map $http_user_agent $rl_key { "~*(Googlebot|Bingbot|DuckDuckBot|Slurp)" ""; default $binary_remote_addr; } limit_req_zone $rl_key zone=crawlers:20m rate=30r/m; limit_conn_zone $binary_remote_addr zone=connip:20m; server { ... location / { limit_conn connip 15; limit_req zone=crawlers burst=60 nodelay; } # Empty UA / known bad bots → 403 map $http_user_agent $badua { default 0; "" 1; "~*(AhrefsBot|MJ12bot|BLEXBot)" 1; } if ($badua) { return 403; } } # ====================================================================That setup alone stops 90 % of “drive‑by” scrapers without hurting the good guys.
Add fail2ban or a CDN in front if it still feels heavy. Happy crawling‑but‑not‑too‑much! 🕷️🛑 - Throttle in Nginx
────────────────────
A. Global IP‑based rate‑limit
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.