Implement a cache hierarchy: CDN→edge→service→DB, keys/TTLs, invalidation, stampede…
- User
Implement a cache hierarchy: CDN→edge→service→DB, keys/TTLs, invalidation, stampede defense.
- Assistant
Below is a practical, implementable cache hierarchy for CDN → edge → service → DB with keying/TTLs, invalidation, and stampede defense. It assumes HTTP APIs returning JSON/HTML plus static assets.
1) Layer responsibilities (what each cache should do)
CDN (global)
- Cache static assets aggressively (immutable).
- Optionally cache public, non-personalized API/HTML responses.
- Provide stale-while-revalidate / stale-if-error to absorb spikes and origin issues.
- Invalidate via purge by tag/key (Fastly surrogate keys, Cloudflare cache tags, Akamai CP codes, etc.).
Edge (regional reverse proxy: NGINX/Varnish/Envoy)
- “Microcache” dynamic responses for 1–10 seconds to collapse bursts.
- Protect origin with request coalescing (one in-flight fill per key).
- Serve stale on upstream errors.
- Invalidate via purge/ban by tag/key (or let TTL handle most).
Service cache (Redis/Memcached + in-process)
- Cache domain objects / aggregates / expensive queries.
- Provide stronger correctness controls than edge/CDN.
- Primary stampede defense: singleflight + distributed lock + soft TTL.
- Invalidate directly on writes (and/or version keys).
DB
- Source of truth.
- Keep queries index-friendly; don’t depend on DB query cache for correctness.
2) Cache keys (normalize + version)
2.1 CDN/edge keys (URL-based)
By default these caches key on:
scheme + host + path + normalized query + selected headers (Vary)
Rules:
- Normalize query params (sorted, remove irrelevant tracking params).
- Avoid caching personalized responses unless you truly segment by auth identity.
Examples:
/v1/products/123?currency=USD&locale=en-US- Keyed with
Vary: Accept-Encoding(and possiblyAccept-Language).
Add a tag header for purge:
Surrogate-Key: product:123 category:9(name depends on CDN)
2.2 Service cache keys (explicit, stable, versioned)
Use a namespace + version + tenant + entity:
{app}:{env}:v{schema}:{tenant}:{entity}:{id}:{variant}Examples:
shop:prod:v3:t42:product:123:locale=en-US:currency=USDshop:prod:v1:t42:categoryProducts:9:page=1:size=50:sort=pop
Versioning:
- Bump
v3 → v4when you change serialization/meaning to avoid mass deletes.
Optional “many-keys” invalidation helper:
- Per-entity version key (no purge needed):
- Data key:
product:123:ver={N} - Version key:
product:123:ver(integer) - Real cache key becomes
product:123:ver=17:locale=en-US
- Data key:
Then on update: increment version; old keys become unreachable.
3) TTL strategy (per layer + soft TTL)
Use hard TTL (absolute expiry) and optionally soft TTL (refresh threshold).
Recommended starting points
Static assets
- CDN:
max-age=31536000, immutable(hash in filename) - Edge: same or bypass (CDN handles it)
- Service: n/a
Public product/category pages (semi-dynamic)
- CDN:
max-age=30+stale-while-revalidate=120+stale-if-error=600 - Edge microcache:
1–5s(even when CDN disabled, edge absorbs bursts) - Service/Redis:
5–30mhard TTL with soft TTL at e.g. 80–90% of hard TTL
Highly dynamic (inventory, pricing)
- CDN: often
no-storeor very smallmax-age=1–5 - Edge:
0–2smicrocache (optional) - Service:
5–60sor compute-specific; consider write-through/invalidation events
Negative caching (404/empty results)
- CDN/edge: 5–30s
- Service: 30–120s (prevents repeated DB hits for missing IDs)
Add jitter everywhere
To prevent synchronized expiry:
ttl = baseTTL ± random(0..baseTTL*0.1)(5–15% jitter is typical)
4) Invalidation model (make it predictable)
You generally want:
- TTL for routine churn
- Event-driven invalidation for correctness on writes
- Versioned keys when fanout purge is expensive
4.1 Service/Redis invalidation (on writes)
Pattern: DB is authoritative, cache is “cache-aside” or “read-through”.
On successful write (after DB commit):
- Delete or version-bump relevant keys:
DEL shop:prod:v3:t42:product:123:*- or
INCR product:123:ver(recommended when variants are many)
- Publish an event:
product.updated(tenant=42, id=123)
4.2 Edge/CDN invalidation
Consume the same event and purge by tag:
- Purge
Surrogate-Key=product:123(also purge related lists like category pages if needed) - If tag purge isn’t available: purge specific URLs (harder), or rely on short TTL + SWR.
Tip: Prefer tag-based purge; URL-based purge doesn’t scale when there are many variants (locale/currency/page).
4.3 List/aggregate invalidation
When
product:123changes, you may need to invalidate:categoryProducts:*for categories containing that product- search indexes / recommendations, etc.
Two common approaches:
- Explicit fanout purge (compute affected category IDs, purge those tags too)
- Version the list keys (e.g.,
category:9:ver) and bump when membership changes
5) Stampede defense (CDN/edge/service)
You want multiple layers of protection.
5.1 CDN
- Use
stale-while-revalidateso CDN serves stale while one request revalidates. - Use
stale-if-errorso CDN shields users during origin issues.
5.2 Edge (NGINX example)
Enable request coalescing for cache fills:
proxy_cache_lock on;proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504;
This ensures only one upstream request populates a missing key; others wait or get stale.
5.3 Service (core protection)
Use:
- In-process singleflight (per instance)
- Distributed lock (Redis) for cross-instance coalescing
- Soft TTL + serve-stale (fast responses even during refresh)
- Negative caching for misses
- Jitter on TTLs
Service read algorithm (pseudocode)
function getProduct(tenant, id, locale, currency): baseKey = "shop:prod:v3:" + tenant + ":product:" + id + ":locale=" + locale + ":currency=" + currency # 1) Fast path: in-memory (optional) v = inProcCache.get(baseKey) if v exists and not expiredHard(v): return v.data # 2) Redis entry = redis.get(baseKey) # entry includes data, hardExp, softExp now = time() if entry exists: if now < entry.softExp: return entry.data if now < entry.hardExp: # Soft-expired: serve stale and refresh in background triggerAsyncRefreshWithCoalescing(baseKey) return entry.data # else hard-expired -> must refresh # 3) Hard miss/expired: singleflight + distributed lock return singleflight.do(baseKey, function(): # re-check Redis inside singleflight (double-check) entry2 = redis.get(baseKey) if entry2 exists and now < entry2.hardExp: return entry2.data lockKey = "lock:" + baseKey token = randomToken() if redis.SET(lockKey, token, NX, PX=5000) == OK: try: data = db.queryProduct(tenant, id, locale, currency) if data missing: # negative cache writeCache(baseKey, NULL, hardTTL=60s, softTTL=30s, jitter=true) return NOT_FOUND writeCache(baseKey, data, hardTTL=10m, softTTL=8m, jitter=true) return data finally: redis.releaseLock(lockKey, token) # Lua compare-and-del else: # Someone else refreshing. Avoid thundering herd: # - brief wait + retry cache sleep(30-80ms) entry3 = redis.get(baseKey) if entry3 exists and now < entry3.hardExp: return entry3.data # last resort: either wait more or hit DB with rate limit return db.queryProduct(...) # guarded by circuit breaker / rate limit )writeCache()should store:datasoftExp = now + softTTLhardExp = now + hardTTL
Notes
- Serve-stale on soft expiry keeps latency low and avoids stampedes.
- Distributed lock TTL must be short and safe (use token + Lua unlock).
- Singleflight prevents N concurrent goroutines/threads in one instance even before Redis lock.
6) HTTP headers to make CDN/edge caching work
For cacheable public responses:
Cache-Control: public, max-age=30, stale-while-revalidate=120, stale-if-error=600ETag(orLast-Modified) so CDN/edge can revalidate cheaplySurrogate-Key: product:123 category:9(or equivalent tagging feature)
For personalized/auth responses:
Cache-Control: private, no-store(unless you intentionally do segmented caching)
For static hashed assets:
Cache-Control: public, max-age=31536000, immutable
7) Write path (end-to-end)
On update (e.g., product 123):
- Write DB transaction
- Publish event
product.updated(tenant=42, id=123, categories=[...]) - Consumers:
- Service:
INCR product:123:ver(or delete relevant Redis keys) - Edge/CDN: purge tags
product:123(andcategory:9etc as needed)
- Service:
If you can’t reliably purge CDN:
- Keep CDN TTL small and rely on
stale-while-revalidate, while service cache remains the main correctness layer.
8) Minimal configuration pointers (edge microcache)
NGINX sketch
proxy_cache_path /var/cache/nginx keys_zone=api:100m max_size=10g inactive=10m use_temp_path=off; server { location /v1/ { proxy_cache api; proxy_cache_valid 200 1s; proxy_cache_valid 404 5s; proxy_cache_lock on; proxy_cache_lock_timeout 2s; proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504; add_header X-Cache $upstream_cache_status; proxy_pass http://service; } }
9) What to measure (so it stays healthy)
- Hit rates per layer (CDN/edge/service)
- Redis:
getlatency, evictions, memory, hot keys - Lock contention rate / wait time
- DB QPS correlated with cache expiry events
- Percent served stale (soft TTL) and refresh success rate
- Purge success/failure and purge latency
If you tell me (a) CDN vendor, (b) edge tech (NGINX/Varnish/Envoy), and (c) service language (Go/Java/Node/Python), I can provide concrete configs + production-ready lock + singleflight code for that stack.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.