Skip to content
TheChatArchiveThe Chat Archive
GPT 5.2· openaiHugging Face Frontier Reasoning Traces

Implement a cache hierarchy: CDN→edge→service→DB, keys/TTLs, invalidation, stampede…

  1. User

    Implement a cache hierarchy: CDN→edge→service→DB, keys/TTLs, invalidation, stampede defense.

  2. Assistant

    Below is a practical, implementable cache hierarchy for CDN → edge → service → DB with keying/TTLs, invalidation, and stampede defense. It assumes HTTP APIs returning JSON/HTML plus static assets.


    1) Layer responsibilities (what each cache should do)

    CDN (global)

    • Cache static assets aggressively (immutable).
    • Optionally cache public, non-personalized API/HTML responses.
    • Provide stale-while-revalidate / stale-if-error to absorb spikes and origin issues.
    • Invalidate via purge by tag/key (Fastly surrogate keys, Cloudflare cache tags, Akamai CP codes, etc.).

    Edge (regional reverse proxy: NGINX/Varnish/Envoy)

    • “Microcache” dynamic responses for 1–10 seconds to collapse bursts.
    • Protect origin with request coalescing (one in-flight fill per key).
    • Serve stale on upstream errors.
    • Invalidate via purge/ban by tag/key (or let TTL handle most).

    Service cache (Redis/Memcached + in-process)

    • Cache domain objects / aggregates / expensive queries.
    • Provide stronger correctness controls than edge/CDN.
    • Primary stampede defense: singleflight + distributed lock + soft TTL.
    • Invalidate directly on writes (and/or version keys).

    DB

    • Source of truth.
    • Keep queries index-friendly; don’t depend on DB query cache for correctness.

    2) Cache keys (normalize + version)

    2.1 CDN/edge keys (URL-based)

    By default these caches key on:

    • scheme + host + path + normalized query + selected headers (Vary)

    Rules:

    • Normalize query params (sorted, remove irrelevant tracking params).
    • Avoid caching personalized responses unless you truly segment by auth identity.

    Examples:

    • /v1/products/123?currency=USD&locale=en-US
    • Keyed with Vary: Accept-Encoding (and possibly Accept-Language).

    Add a tag header for purge:

    • Surrogate-Key: product:123 category:9 (name depends on CDN)

    2.2 Service cache keys (explicit, stable, versioned)

    Use a namespace + version + tenant + entity:

    {app}:{env}:v{schema}:{tenant}:{entity}:{id}:{variant}
    

    Examples:

    • shop:prod:v3:t42:product:123:locale=en-US:currency=USD
    • shop:prod:v1:t42:categoryProducts:9:page=1:size=50:sort=pop

    Versioning:

    • Bump v3 → v4 when you change serialization/meaning to avoid mass deletes.

    Optional “many-keys” invalidation helper:

    • Per-entity version key (no purge needed):
      • Data key: product:123:ver={N}
      • Version key: product:123:ver (integer)
      • Real cache key becomes product:123:ver=17:locale=en-US

    Then on update: increment version; old keys become unreachable.


    3) TTL strategy (per layer + soft TTL)

    Use hard TTL (absolute expiry) and optionally soft TTL (refresh threshold).

    Recommended starting points

    Static assets

    • CDN: max-age=31536000, immutable (hash in filename)
    • Edge: same or bypass (CDN handles it)
    • Service: n/a

    Public product/category pages (semi-dynamic)

    • CDN: max-age=30 + stale-while-revalidate=120 + stale-if-error=600
    • Edge microcache: 1–5s (even when CDN disabled, edge absorbs bursts)
    • Service/Redis: 5–30m hard TTL with soft TTL at e.g. 80–90% of hard TTL

    Highly dynamic (inventory, pricing)

    • CDN: often no-store or very small max-age=1–5
    • Edge: 0–2s microcache (optional)
    • Service: 5–60s or compute-specific; consider write-through/invalidation events

    Negative caching (404/empty results)

    • CDN/edge: 5–30s
    • Service: 30–120s (prevents repeated DB hits for missing IDs)

    Add jitter everywhere

    To prevent synchronized expiry:

    • ttl = baseTTL ± random(0..baseTTL*0.1) (5–15% jitter is typical)

    4) Invalidation model (make it predictable)

    You generally want:

    • TTL for routine churn
    • Event-driven invalidation for correctness on writes
    • Versioned keys when fanout purge is expensive

    4.1 Service/Redis invalidation (on writes)

    Pattern: DB is authoritative, cache is “cache-aside” or “read-through”.

    On successful write (after DB commit):

    • Delete or version-bump relevant keys:
      • DEL shop:prod:v3:t42:product:123:*
      • or INCR product:123:ver (recommended when variants are many)
    • Publish an event: product.updated(tenant=42, id=123)

    4.2 Edge/CDN invalidation

    Consume the same event and purge by tag:

    • Purge Surrogate-Key=product:123 (also purge related lists like category pages if needed)
    • If tag purge isn’t available: purge specific URLs (harder), or rely on short TTL + SWR.

    Tip: Prefer tag-based purge; URL-based purge doesn’t scale when there are many variants (locale/currency/page).

    4.3 List/aggregate invalidation

    When product:123 changes, you may need to invalidate:

    • categoryProducts:* for categories containing that product
    • search indexes / recommendations, etc.

    Two common approaches:

    1. Explicit fanout purge (compute affected category IDs, purge those tags too)
    2. Version the list keys (e.g., category:9:ver) and bump when membership changes

    5) Stampede defense (CDN/edge/service)

    You want multiple layers of protection.

    5.1 CDN

    • Use stale-while-revalidate so CDN serves stale while one request revalidates.
    • Use stale-if-error so CDN shields users during origin issues.

    5.2 Edge (NGINX example)

    Enable request coalescing for cache fills:

    • proxy_cache_lock on;
    • proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504;

    This ensures only one upstream request populates a missing key; others wait or get stale.

    5.3 Service (core protection)

    Use:

    1. In-process singleflight (per instance)
    2. Distributed lock (Redis) for cross-instance coalescing
    3. Soft TTL + serve-stale (fast responses even during refresh)
    4. Negative caching for misses
    5. Jitter on TTLs

    Service read algorithm (pseudocode)

    function getProduct(tenant, id, locale, currency):
      baseKey = "shop:prod:v3:" + tenant + ":product:" + id + ":locale=" + locale + ":currency=" + currency
    
      # 1) Fast path: in-memory (optional)
      v = inProcCache.get(baseKey)
      if v exists and not expiredHard(v): return v.data
    
      # 2) Redis
      entry = redis.get(baseKey)  # entry includes data, hardExp, softExp
      now = time()
    
      if entry exists:
        if now < entry.softExp:
          return entry.data
        if now < entry.hardExp:
          # Soft-expired: serve stale and refresh in background
          triggerAsyncRefreshWithCoalescing(baseKey)
          return entry.data
        # else hard-expired -> must refresh
    
      # 3) Hard miss/expired: singleflight + distributed lock
      return singleflight.do(baseKey, function():
        # re-check Redis inside singleflight (double-check)
        entry2 = redis.get(baseKey)
        if entry2 exists and now < entry2.hardExp: return entry2.data
    
        lockKey = "lock:" + baseKey
        token = randomToken()
        if redis.SET(lockKey, token, NX, PX=5000) == OK:
          try:
            data = db.queryProduct(tenant, id, locale, currency)
            if data missing:
              # negative cache
              writeCache(baseKey, NULL, hardTTL=60s, softTTL=30s, jitter=true)
              return NOT_FOUND
            writeCache(baseKey, data, hardTTL=10m, softTTL=8m, jitter=true)
            return data
          finally:
            redis.releaseLock(lockKey, token)  # Lua compare-and-del
        else:
          # Someone else refreshing. Avoid thundering herd:
          # - brief wait + retry cache
          sleep(30-80ms)
          entry3 = redis.get(baseKey)
          if entry3 exists and now < entry3.hardExp: return entry3.data
          # last resort: either wait more or hit DB with rate limit
          return db.queryProduct(...)  # guarded by circuit breaker / rate limit
      )
    

    writeCache() should store:

    • data
    • softExp = now + softTTL
    • hardExp = now + hardTTL

    Notes

    • Serve-stale on soft expiry keeps latency low and avoids stampedes.
    • Distributed lock TTL must be short and safe (use token + Lua unlock).
    • Singleflight prevents N concurrent goroutines/threads in one instance even before Redis lock.

    6) HTTP headers to make CDN/edge caching work

    For cacheable public responses:

    • Cache-Control: public, max-age=30, stale-while-revalidate=120, stale-if-error=600
    • ETag (or Last-Modified) so CDN/edge can revalidate cheaply
    • Surrogate-Key: product:123 category:9 (or equivalent tagging feature)

    For personalized/auth responses:

    • Cache-Control: private, no-store (unless you intentionally do segmented caching)

    For static hashed assets:

    • Cache-Control: public, max-age=31536000, immutable

    7) Write path (end-to-end)

    On update (e.g., product 123):

    1. Write DB transaction
    2. Publish event product.updated(tenant=42, id=123, categories=[...])
    3. Consumers:
      • Service: INCR product:123:ver (or delete relevant Redis keys)
      • Edge/CDN: purge tags product:123 (and category:9 etc as needed)

    If you can’t reliably purge CDN:

    • Keep CDN TTL small and rely on stale-while-revalidate, while service cache remains the main correctness layer.

    8) Minimal configuration pointers (edge microcache)

    NGINX sketch

    proxy_cache_path /var/cache/nginx keys_zone=api:100m max_size=10g inactive=10m use_temp_path=off;
    
    server {
      location /v1/ {
        proxy_cache api;
        proxy_cache_valid 200 1s;
        proxy_cache_valid 404 5s;
    
        proxy_cache_lock on;
        proxy_cache_lock_timeout 2s;
        proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504;
    
        add_header X-Cache $upstream_cache_status;
        proxy_pass http://service;
      }
    }
    

    9) What to measure (so it stays healthy)

    • Hit rates per layer (CDN/edge/service)
    • Redis: get latency, evictions, memory, hot keys
    • Lock contention rate / wait time
    • DB QPS correlated with cache expiry events
    • Percent served stale (soft TTL) and refresh success rate
    • Purge success/failure and purge latency

    If you tell me (a) CDN vendor, (b) edge tech (NGINX/Varnish/Envoy), and (c) service language (Go/Java/Node/Python), I can provide concrete configs + production-ready lock + singleflight code for that stack.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.