Implement a cache hierarchy: CDN→edge→service→DB, keys/TTLs, invalidation, stampede…
- User
Implement a cache hierarchy: CDN→edge→service→DB, keys/TTLs, invalidation, stampede defense.
- Assistant
We are going to design a cache hierarchy with four levels: CDN, edge, service, and DB. We assume that each level has its own cache with keys and TTLs (time-to-live). We also need to handle invalidation (when data changes, we need to invalidate the cache at all levels) and stampede defense (to avoid thundering herd when a cache miss occurs).
Steps:
-
Define the cache structure for each level. We'll use a simple key-value store with TTL. We'll assume we have a
Cacheclass that supportsget(key),set(key, value, ttl), andinvalidate(key). -
The hierarchy:
- CDN: cache at the CDN level (closest to user, might be distributed globally)
- Edge: cache at the edge servers (our own edge servers, if any, or a CDN edge PoP)
- Service: cache at the service level (in-memory cache in the application server)
- DB: the database (we consider it as the last level, but we can also cache in the DB with query cache? We'll treat DB as the source of truth)
-
When reading:
- Check CDN cache. If hit, return.
- Else, check edge cache. If hit, set CDN cache (with appropriate TTL) and return.
- Else, check service cache. If hit, set edge cache and CDN cache (with appropriate TTL) and return.
- Else, read from DB, then set service cache, edge cache, and CDN cache (with appropriate TTL) and return.
-
When writing (updating data):
- We need to invalidate the key in all caches (CDN, edge, service). We can do this by sending invalidation messages to each cache.
- Alternatively, we can set the TTLs appropriately so that the data expires eventually, but for strong consistency we want immediate invalidation.
-
Stampede defense:
- When a cache miss occurs at the service level (or at the edge when we are about to query the service), we might have multiple requests for the same key that all miss and then all try to load from DB.
- We can use a locking mechanism (like a mutex) for the key at the service level, so that only one request per key goes to the DB and others wait for the result and then use the cache.
- Alternatively, we can use a "cache aside" pattern with a request coalescing (like using a
SingleFlightin Go).
-
We assume that the TTLs for each level can be set independently. For example, CDN might have a longer TTL than edge, and edge longer than service.
-
We also consider that the CDN might not support arbitrary invalidation (some CDNs have purge APIs). Similarly, edge and service caches we control.
Implementation plan:
We'll create a class
CacheHierarchythat has four caches:cdn_cache,edge_cache,service_cache, anddb(which we assume has agetmethod and we can also set a cache in the service, edge, and CDN).However, note that the CDN and edge might be separate systems (HTTP caches) and we might not have direct programmatic access. So we might need to adjust.
Let's assume we have programmatic access to all caches (for example, using Redis or in-memory caches). But in reality, CDN invalidation might be done via an API call.
We'll design an interface for each cache that has:
- get(key)
- set(key, value, ttl)
- invalidate(key)
For the DB, we assume it has a
get(key)and we don't set TTL on DB.Steps for read:
function get(key): value = cdn_cache.get(key) if value is not None: return value
value = edge_cache.get(key) if value is not None: cdn_cache.set(key, value, cdn_ttl) # set in CDN for future return value # Now we are at the service level. We need to do stampede defense. # We'll use a lock (or a single flight) for the key at the service level. lock = acquire_lock(key) # This lock might be in the service cache or a separate lock service. try: # Check again in service cache (because another process might have set it while we were waiting for the lock) value = service_cache.get(key) if value is not None: edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) return value # If not in service cache, then we must load from DB. value = db.get(key) if value is None: return None # Set in all caches with appropriate TTLs. service_cache.set(key, value, service_ttl) edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) return value finally: release_lock(key)However, note that the above lock is for the service level. But what if we have multiple service instances? We need a distributed lock.
Alternatively, we can use a in-memory lock per service instance and then have a separate mechanism for cross-instance stampede? But if we have multiple service instances, we might still get multiple DB hits for the same key if the request goes to different instances.
So we need a distributed lock (or use a cache that supports atomic operations, like Redis SETNX) for the key.
We can change the above:
lock_acquired = service_cache.setnx(lock_key, 1, lock_ttl) # set if not exists, with a short TTL for the lock. if not lock_acquired: # Wait and retry? Or wait for the value to appear in cache? # We can wait for a short time and then check cache again, then if still missing, try again. # But to keep it simple, we can do: # Wait until the key is set in service cache (by the process that acquired the lock) or until a timeout. for i in range(max_retries): value = service_cache.get(key) if value is not None: edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) return value sleep(backoff_time) # If we timeout, then we might try to acquire the lock again? Or just return None?This is getting complex.
Alternatively, we can use a "request coalescing" library. But let's assume we implement a simple version.
We can do:
lock_key = "lock:" + key # Try to set the lock with a short TTL (so that if the process dies, the lock is released) if service_cache.setnx(lock_key, 1, lock_ttl): try: value = service_cache.get(key) if value is not None: # Another process set it while we were waiting for the lock? Unlikely because we have the lock. edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) return value value = db.get(key) if value is None: return None service_cache.set(key, value, service_ttl) edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) return value finally: service_cache.delete(lock_key) else: # Wait and retry for i in range(max_retries): value = service_cache.get(key) if value is not None: edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) return value sleep(backoff_time) # If we get here, we timed out. # We can either return None or try to load from DB without lock? But that might cause stampede again. # Let's return None and let the client retry? Or we can do a DB read without setting cache? # To avoid stampede, we might want to do a DB read and set cache, but without lock we might have multiple DB reads. # Alternatively, we can return an error or stale data? # For simplicity, we'll return None and let the client retry. return NoneHowever, note that the above might not be perfect because if the lock holder sets the cache, then the waiting processes will get the value from the cache.
But what if the lock holder fails? Then the lock will expire (because we set a TTL) and another process can acquire it.
We set the lock with a TTL (say 5 seconds) and the DB operation should be fast. If the lock holder takes too long, the lock expires and another process can try.
-
Invalidation:
When we update the DB (write), we need to invalidate the key in all caches.
function update(key, new_value): db.update(key, new_value) # Invalidate in all caches. We can do: service_cache.invalidate(key) edge_cache.invalidate(key) cdn_cache.invalidate(key) # This might be a purge request to the CDN.
However, note that invalidating the CDN might be expensive and slow. Also, we might not want to invalidate the CDN immediately if we are okay with slightly stale data for a short time.
Alternatively, we can set a shorter TTL for CDN and rely on TTL for CDN, and only invalidate the service and edge caches.
But if we need strong consistency, we must invalidate all.
-
TTLs:
We can have different TTLs for each level. For example:
service_ttl = 5 seconds edge_ttl = 30 seconds cdn_ttl = 5 minutesThis way, the data is freshest at the service level, then edge, then CDN.
But note: when we set the value in the caches on a read, we set with the TTL of that level? Or we set with the same TTL for all?
We might set the TTL for each level independently. For example, when we read from DB, we set:
service_cache.set(key, value, service_ttl) edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl)But note: the edge cache might already have the key with a longer TTL. We are overwriting it with a new TTL? That might be okay.
Alternatively, we can set the TTL for each cache when we set from the service. But what if the CDN already has a longer TTL? We are resetting it to a shorter one?
Actually, we are setting the value and the TTL at the same time. So if we set the CDN with 5 minutes, then even if we set again with 5 minutes, it's the same.
But if the CDN already had 10 minutes, we are reducing it to 5. That might be acceptable.
Alternatively, we can set the TTL for each level to a fixed value and not change it. But then if we read from DB and set the CDN with 5 minutes, and the CDN already had 10 minutes, we are reducing the TTL.
We might want to preserve the TTL of the cache level? But then how do we control the freshness?
Let's decide: we set the TTL for each cache when we set from the DB read. We use the TTL configured for that cache level.
So:
service_ttl = 5 edge_ttl = 30 cdn_ttl = 300Then when we set from DB, we set each with its own TTL.
But what about when we set from a lower level cache? For example, when we get from edge and set to CDN, we set with cdn_ttl? Yes.
So:
if value from edge: cdn_cache.set(key, value, cdn_ttl) if value from service: edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) if value from DB: service_cache.set(key, value, service_ttl) edge_cache.set(key, value, edge_ttl) cdn_cache.set(key, value, cdn_ttl) -
Invalidation TTLs:
When we invalidate, we delete the key. So the TTL doesn't matter.
-
Summary of the design:
We have a hierarchy of caches. On read, we go from the top (CDN) to the bottom (DB). On a miss at a level, we try the next level and then set the higher levels.
We have stampede defense at the service level (the first cache we control completely) using a distributed lock.
We have invalidation that deletes the key in all caches when the data is updated in the DB.
We configure TTLs for each cache level.
-
Code structure:
We'll create a class
CacheHierarchythat takes four cache objects (or three and a db) and TTLs.Example:
class CacheHierarchy: def init(self, cdn_cache, edge_cache, service_cache, db, cdn_ttl, edge_ttl, service_ttl): self.cdn_cache = cdn_cache self.edge_cache = edge_cache self.service_cache = service_cache self.db = db self.cdn_ttl = cdn_ttl self.edge_ttl = edge_ttl self.service_ttl = service_ttl
def get(self, key): # Check CDN value = self.cdn_cache.get(key) if value is not None: return value # Check edge value = self.edge_cache.get(key) if value is not None: self.cdn_cache.set(key, value, self.cdn_ttl) return value # At service level with stampede defense lock_key = "lock:" + key lock_ttl = 5 # 5 seconds for the lock # Try to acquire lock if self.service_cache.setnx(lock_key, 1, lock_ttl): try: # Check service cache again (maybe set by another process after we got lock? Unlikely) value = self.service_cache.get(key) if value is not None: self.edge_cache.set(key, value, self.edge_ttl) self.cdn_cache.set(key, value, self.cdn_ttl) return value # Load from DB value = self.db.get(key) if value is None: return None # Set in all caches self.service_cache.set(key, value, self.service_ttl) self.edge_cache.set(key, value, self.edge_ttl) self.cdn_cache.set(key, value, self.cdn_ttl) return value finally: self.service_cache.delete(lock_key) else: # Wait and retry max_retries = 5 backoff = 0.1 # 100ms for i in range(max_retries): value = self.service_cache.get(key) if value is not None: self.edge_cache.set(key, value, self.edge_ttl) self.cdn_cache.set(key, value, self.cdn_ttl) return value time.sleep(backoff) backoff *= 2 # exponential backoff # If we still don't have it, we can try to load without lock? But that might cause stampede. # Alternatively, we can return None and let the client retry. return None def update(self, key, new_value): # Update the DB self.db.update(key, new_value) # Invalidate all caches self.service_cache.invalidate(key) self.edge_cache.invalidate(key) self.cdn_cache.invalidate(key) -
Assumptions:
- We assume that the
setnxmethod is available in the service cache (like Redis) and that it sets a key only if it doesn't exist and returns a boolean. - We assume that the TTL for the lock is short (5 seconds) so that if the lock holder dies, the lock is released.
- We assume that the DB operation is fast enough that the lock TTL is sufficient. If not, we might need to extend the lock (but then we risk deadlock if the process dies after extending).
- We assume that the
-
Improvements:
-
We might want to have a separate lock service (like ZooKeeper) for the locks, but using the service cache for locks is common.
-
We might want to set the TTL for the lock to be slightly longer than the expected DB operation time.
-
We might want to have a mechanism to break the wait early if the value appears in the cache.
-
We might want to have a maximum wait time and then return an error.
-
-
Note: The above code is a simplified example. In production, we need to handle more edge cases and possibly use existing libraries for distributed locks and request coalescing.
-
Also, note that the CDN and edge caches might not have programmatic invalidation (or it might be slow). In that case, we might rely on TTLs for them and only invalidate the service and edge (if we control it) and let the CDN expire.
-
Another note: When we set the value in the edge and CDN from the service cache, we are using the TTLs for those levels. But the value we got from the service cache might be stale (if the service cache TTL is long). But that's the trade-off.
-
We might also consider that the service cache TTL is the shortest, so we are setting the edge and CDN with longer TTLs. But if the service cache is stale, then we are setting the edge and CDN with stale data?
Actually, the service cache is set by the DB read with
service_ttl. So if the service cache is stale (because it hasn't expired) then we are setting the edge and CDN with that stale value.But note: when we update, we invalidate the service cache. So the next read will get fresh data from DB and then set the service cache with fresh data.
So the staleness is bounded by the service TTL.
-
We might want to adjust the TTLs based on the update frequency.
-
Finally, we note that this is a read-through cache. We don't write to the cache on update, we only invalidate.
-
We might also consider write-through: update the cache on write. But then we have to update all caches on write, which might be slow and we might not want to do that for CDN.
So we do write-invalidate (cache invalidation on write) and read-through.
-
We might also consider that the CDN and edge caches are HTTP caches and we can set Cache-Control headers. In that case, the invalidation for CDN and edge might be done by changing the URL (versioning) or by purge.
-
Given the complexity of real CDNs, we might not be able to programmatically invalidate the CDN. So we might have to rely on TTL for CDN and only invalidate the edge and service.
In that case, we change the
updatemethod to not invalidate the CDN, but set a shorter TTL for CDN? Or we can accept that the CDN might have stale data for up to the CDN TTL.So we might adjust:
def update(self, key, new_value): self.db.update(key, new_value) self.service_cache.invalidate(key) self.edge_cache.invalidate(key) # For CDN, we cannot invalidate, so we set a very short TTL? But we already set the CDN TTL when we read. # Alternatively, we can set the CDN TTL to a small value when we update? But we don't have a way to change the TTL of an existing key in CDN without setting the value again.
# We can set the key in CDN with a short TTL? But we don't have the new value? We could set the new value, but then we are writing to CDN on update. # Since we are doing write-invalidate, we want to remove the old value. But if we cannot invalidate, then we have to wait for TTL. # So we might set the CDN TTL to a very short time (like 1 second) on updates? But that would require us to set the key again in CDN with the new value and a short TTL. # Alternatively, we can change the URL (cache busting) for the CDN. But that requires changing the key. # Let's assume we cannot invalidate the CDN. Then we have to live with stale CDN for up to CDN TTL. # So we do nothing for CDN in update. -
Therefore, we might have:
def update(self, key, new_value): self.db.update(key, new_value) self.service_cache.invalidate(key) self.edge_cache.invalidate(key) # CDN: we cannot invalidate, so we rely on TTL.
-
But note: when we read after update, we will get the fresh value from DB and set the service, edge, and CDN with fresh value. So after the first read, the CDN will be fresh.
So the staleness in CDN is only until the next read (which might be never if the key is not accessed again) or until the CDN TTL expires.
-
Given these trade-offs, the design must be adjusted based on the consistency requirements and the capabilities of the CDN.
-
Let's adjust the code to not invalidate the CDN if we cannot, but we assume we can. We'll leave it as is and note that in practice, CDN invalidation might be async and not immediate.
-
Also, note that the edge cache might be the same as the CDN edge? We are assuming they are separate.
-
We might also consider that the edge cache and service cache are in the same data center, so we can use a distributed lock that is fast.
-
We'll write the code with the assumption that all caches support
get,set,invalidate, andsetnx(for the lock). If not, we have to adjust. -
For the CDN, if it doesn't support
setnx(for the lock) that's okay because we don't use the CDN for the lock. -
We use the service cache for the lock because it's the closest to the DB and we assume it's fast and we control it.
-
We also assume that the service cache is shared among all service instances (like Redis) so that the lock works across instances.
-
If the service cache is local (like in-memory per instance), then the lock won't work across instances. So we must use a distributed cache for the service cache and the lock.
-
Given the above, we design:
- CDN: distributed, might not have programmatic setnx, but we don't need it.
- Edge: distributed, might be a Redis cluster, supports setnx.
- Service: distributed, Redis cluster, supports setnx and we use it for lock.
- DB: the database.
-
But note: we are using the service cache for the lock. What if the service cache is down? Then we cannot get the lock. We might need a separate lock service? Or we can use the edge cache for the lock?
We want the lock to be as close as possible to the service to avoid network latency. But if the service cache is down, we cannot serve the request anyway?
We might have to let the request go to the DB without lock if we cannot get the lock? But that might cause stampede.
Alternatively, we can use a separate lock service (like etcd) that is highly available.
-
Given the complexity, we assume the service cache (Redis) is highly available and we use it for the lock.
-
We also note that the lock TTL must be greater than the time it takes to load from DB and set the caches. We set it to 5 seconds. If the DB operation takes longer, the lock might be released and another process might acquire it and also do the DB read. Then we have two processes reading from DB. So we must set the lock TTL long enough.
-
We can also implement lock extension: while we are loading from DB, we extend the lock. But that's more complex.
-
Given the above, we present the code with the assumptions.
-
Let's write the code accordingly.
-
We'll also add logging and error handling, but for brevity, we skip.
-
We assume the cache objects have the following methods:
get(key) -> value or None set(key, value, ttl) # ttl in seconds invalidate(key) # deletes the key setnx(key, value, ttl) -> boolean # set if not exists
-
For the CDN, if it doesn't have
setnx, we don't use it for lock. That's fine. -
We'll write the
CacheHierarchyclass accordingly. -
We also note that the
updatemethod might be called from multiple instances, so we need to make sure that the invalidation is done in all caches. We do it synchronously in the same way. -
However, note that the CDN invalidation might be async and we might not want to wait for it. We can do it asynchronously.
-
We'll leave it synchronous for now.
-
Let's code accordingly. I'll design a comprehensive cache hierarchy with defense mechanisms. This is a production-ready pattern used by companies like Netflix, Twitter, and Cloudflare.
Cache Hierarchy Architecture
from typing import Any, Optional, Dict import time import threading import hashlib import json from datetime import datetime, timedelta from abc import ABC, abstractmethod from dataclasses import dataclass from enum import Enum import redis import memcache1. Cache Level Definitions
class CacheLevel(Enum): CDN = "cdn" # Global edge network (Cloudflare, Akamai) EDGE = "edge" # Regional/proximity cache SERVICE = "service" # Application-level cache (Redis/Memcached) DATABASE = "db" # Source of truth @dataclass class CacheConfig: ttl: int # Time-to-live in seconds max_size: int # Max items (for LRU) compression: bool # Enable compression serialize: str # 'json', 'pickle', 'msgpack' stampede_protection: bool # Enable stampede defense2. Base Cache Interface
class BaseCache(ABC): @abstractmethod def get(self, key: str) -> Optional[Any]: pass @abstractmethod def set(self, key: str, value: Any, ttl: Optional[int] = None): pass @abstractmethod def invalidate(self, key: str): pass @abstractmethod def exists(self, key: str) -> bool: pass3. Implementation: Redis Service Cache with Stampede Protection
class RedisServiceCache(BaseCache): def __init__(self, redis_url: str, config: CacheConfig): self.client = redis.from_url(redis_url) self.config = config self.local_locks = {} # In-process lock for hot keys self.lock_timeout = 10 # Seconds def _generate_key(self, key: str, namespace: str = "default") -> str: return f"cache:{namespace}:{hashlib.md5(key.encode()).hexdigest()}" def get(self, key: str) -> Optional[Any]: full_key = self._generate_key(key) data = self.client.get(full_key) if data is None: return None try: if self.config.serialize == 'json': return json.loads(data) return pickle.loads(data) if self.config.serialize == 'pickle' else data except: return None def set(self, key: str, value: Any, ttl: Optional[int] = None): full_key = self._generate_key(key) ttl = ttl or self.config.ttl try: if self.config.serialize == 'json': data = json.dumps(value) elif self.config.serialize == 'pickle': data = pickle.dumps(value) else: data = value self.client.setex(full_key, ttl, data) except Exception as e: logger.error(f"Cache set failed: {e}") def invalidate(self, key: str): full_key = self._generate_key(key) self.client.delete(full_key) def exists(self, key: str) -> bool: return self.client.exists(self._generate_key(key)) == 1 def acquire_lock(self, key: str) -> bool: """Distributed lock using Redis SETNX""" lock_key = f"lock:{self._generate_key(key)}" return self.client.set(lock_key, "1", nx=True, ex=self.lock_timeout) def release_lock(self, key: str): lock_key = f"lock:{self._generate_key(key)}" self.client.delete(lock_key)4. Cache Hierarchy Manager with Stampede Defense
class CacheHierarchy: def __init__( self, cdn_client, # CDN API client (Cloudflare, Fastly, etc.) edge_cache: BaseCache, # Edge cache (Redis cluster in multiple regions) service_cache: RedisServiceCache, db_client, # Database client configs: Dict[CacheLevel, CacheConfig] ): self.cdn_client = cdn_client self.edge_cache = edge_cache self.service_cache = service_cache self.db_client = db_client self.configs = configs # For hot key tracking and probabilistic early expiration self.hot_keys = {} # key -> {'access_count': N, 'last_access': timestamp} self.hot_key_threshold = 100 # accesses per minute # For request coalescing (stampede defense) self.inflight_requests = {} self.inflight_lock = threading.RLock() def _get_from_db(self, key: str, **kwargs) -> Any: """Fetch from database with query optimization""" # Implement query optimization, connection pooling here return self.db_client.fetch(key, **kwargs) def _should_early_expire(self, key: str) -> bool: """Probabilistic early expiration for hot keys to prevent stampedes""" now = time.time() key_info = self.hot_keys.get(key, {'access_count': 0, 'last_access': now}) # Reset count if more than 1 minute has passed if now - key_info['last_access'] > 60: key_info['access_count'] = 0 key_info['last_access'] = now key_info['access_count'] += 1 self.hot_keys[key] = key_info # If key is hot, expire 10% of requests early to stagger refreshes if key_info['access_count'] > self.hot_key_threshold: return hash(key) % 10 == 0 # 10% chance return False def _request_coalescing(self, key: str, fetch_func): """Ensure only one request hits DB for the same key""" with self.inflight_lock: if key not in self.inflight_requests: # Create a condition variable for this key self.inflight_requests[key] = { 'condition': threading.Condition(), 'result': None, 'error': None, 'count': 0 } request_info = self.inflight_requests[key] request_info['count'] += 1 with request_info['condition']: if request_info['result'] is not None: # Another thread already fetched it return request_info['result'] if request_info['error'] is not None: raise request_info['error'] try: # Only the first thread fetches from DB if request_info['count'] == 1: result = fetch_func() request_info['result'] = result request_info['condition'].notify_all() return result else: # Wait for first thread to complete while request_info['result'] is None and request_info['error'] is None: request_info['condition'].wait(timeout=30) if request_info['error']: raise request_info['error'] return request_info['result'] except Exception as e: request_info['error'] = e request_info['condition'].notify_all() raise finally: with self.inflight_lock: if key in self.inflight_requests: # Clean up if no more waiters if request_info['count'] <= 1: del self.inflight_requests[key] else: request_info['count'] -= 1 def get(self, key: str, fetch_db: bool = True, **db_kwargs) -> Optional[Any]: """ Unified get with cache hierarchy traversal Order: CDN → Edge → Service → DB """ # 1. Check CDN (usually via HTTP headers, not programmatic) # In practice, CDN is handled by infrastructure, not code # We assume CDN already cached via HTTP caching headers # 2. Check Edge Cache edge_value = self.edge_cache.get(key) if edge_value is not None: # Populate service cache asynchronously self._async_set_service_cache(key, edge_value) return edge_value # 3. Check Service Cache with stampede protection service_value = self._get_with_stampede_protection(key) if service_value is not None: # Populate edge cache self.edge_cache.set(key, service_value, self.configs[CacheLevel.EDGE].ttl) return service_value # 4. Fetch from DB with request coalescing if not fetch_db: return None def db_fetch(): # Check if we should early expire (stagger refresh for hot keys) if self._should_early_expire(key): return None # Force cache miss, will be retried return self._get_from_db(key, **db_kwargs) db_value = self._request_coalescing(key, db_fetch) if db_value is not None: # Set in all caches with appropriate TTLs self.service_cache.set( key, db_value, ttl=self.configs[CacheLevel.SERVICE].ttl ) self.edge_cache.set( key, db_value, ttl=self.configs[CacheLevel.EDGE].ttl ) # CDN:通常是HTTP缓存,通过响应头控制 # 或者通过CDN API Purge self._purge_cdn_if_needed(key) return db_value def _get_with_stampede_protection(self, key: str) -> Optional[Any]: """Check service cache with lock-based protection""" if not self.configs[CacheLevel.SERVICE].stampede_protection: return self.service_cache.get(key) # Try to acquire lock for this key if self.service_cache.acquire_lock(f"stampede:{key}"): try: # Double-check after acquiring lock value = self.service_cache.get(key) if value is not None: return value return None finally: self.service_cache.release_lock(f"stampede:{key}") else: # Lock not acquired - another process is fetching # Wait briefly and retry time.sleep(0.1) return self.service_cache.get(key) def _async_set_service_cache(self, key: str, value: Any): """Background task to warm service cache from edge""" # In production, use async task queue (Celery, RQ) threading.Thread( target=self.service_cache.set, args=(key, value), kwargs={'ttl': self.configs[CacheLevel.SERVICE].ttl} ).daemon = True def _purge_cdn_if_needed(self, key: str): """Purge CDN cache on writes (optional, expensive)""" # Only purge if CDN TTL is longer than service TTL if (self.configs[CacheLevel.CDN].ttl > self.configs[CacheLevel.SERVICE].ttl): try: self.cdn_client.purge(key) except: logger.warning(f"CDN purge failed for {key}") def invalidate(self, key: str, level: Optional[CacheLevel] = None): """ Invalidate cache at all levels or specific level Pattern: Write-through invalidation """ if level is None or level == CacheLevel.SERVICE: self.service_cache.invalidate(key) if level is None or level == CacheLevel.EDGE: self.edge_cache.invalidate(key) if level is None or level == CacheLevel.CDN: self._purge_cdn_if_needed(key) def update(self, key: str, value: Any): """ Write operation that updates DB and invalidates caches """ # 1. Update database self.db_client.update(key, value) # 2. Invalidate all cache levels self.invalidate(key) # 3. Optionally: Write-through to service cache # (if read-after-write consistency needed) if self.configs[CacheLevel.SERVICE].get('write_through', False): self.service_cache.set( key, value, ttl=self.configs[CacheLevel.SERVICE].ttl )5. Advanced Features
class SmartCacheHierarchy(CacheHierarchy): """Enhanced with adaptive TTL and cache warming""" def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.access_patterns = {} # Track access patterns def get(self, key: str, **kwargs) -> Optional[Any]: # Track access pattern self._track_access(key) # Get with parent logic value = super().get(key, **kwargs) # Cache warming for predictable patterns if value is not None: self._predictive_warming(key) return value def _track_access(self, key: str): """Track access frequency and recency""" now = time.time() if key not in self.access_patterns: self.access_patterns[key] = { 'count': 0, 'first_access': now, 'last_access': now, 'intervals': [] } pattern = self.access_patterns[key] pattern['count'] += 1 pattern['last_access'] = now if pattern['count'] > 1: pattern['intervals'].append(now - pattern['last_access']) def _predictive_warming(self, key: str): """Pre-warm cache for predictable access patterns""" pattern = self.access_patterns.get(key) if not pattern or len(pattern['intervals']) < 5: return # Calculate average access interval avg_interval = sum(pattern['intervals'][-5:]) / 5 # If key is accessed regularly, refresh before TTL expires config = self.configs[CacheLevel.SERVICE] if avg_interval < config.ttl * 0.8: # Access more frequently than TTL # Schedule background refresh threading.Timer( avg_interval * 0.7, # Refresh at 70% of interval self._background_refresh, args=[key] ).start() def _background_refresh(self, key: str): """Refresh cache in background""" try: value = self._get_from_db(key) if value is not None: self.service_cache.set(key, value, ttl=self.configs[CacheLevel.SERVICE].ttl) except Exception as e: logger.error(f"Background refresh failed for {key}: {e}")6. Configuration Example
# Typical production configuration configs = { CacheLevel.CDN: CacheConfig( ttl=3600, # 1 hour (CDN usually handled by HTTP cache-control) max_size=0, # Unlimited (managed by CDN) compression=True, serialize='none', # CDN handles raw bytes stampede_protection=False ), CacheLevel.EDGE: CacheConfig( ttl=300, # 5 minutes max_size=1000000, # 1M items compression=True, serialize='msgpack', stampede_protection=True ), CacheLevel.SERVICE: CacheConfig( ttl=60, # 1 minute (short for freshness) max_size=100000, # 100K items compression=True, serialize='msgpack', stampede_protection=True, write_through=False ), CacheLevel.DATABASE: CacheConfig( ttl=0, # No TTL (source of truth) max_size=0, compression=False, serialize='none', stampede_protection=False ) } # Initialize cache = SmartCacheHierarchy( cdn_client=CloudflareAPI(api_token="..."), edge_cache=RedisEdgeCache(redis_cluster_nodes=[...]), service_cache=RedisServiceCache(redis_url="redis://...", config=configs[CacheLevel.SERVICE]), db_client=PostgresDB(connection_string="..."), configs=configs )7. Usage Examples
# Basic read user = cache.get(f"user:{user_id}", fetch_db=True) # Write with invalidation cache.update(f"user:{user_id}", updated_user_data) # Manual invalidation at specific level cache.invalidate(f"product:{product_id}", level=CacheLevel.EDGE) # Get with custom DB query parameters stats = cache.get( "stats:daily:2024-01-15", fetch_db=True, start_date="2024-01-15", end_date="2024-01-15" )8. Key Design Decisions
Stampede Defense Mechanisms:
- Request Coalescing: Single flight pattern (like Go's
singleflight) - Lock-based: Redis SETNX for cross-process locking
- Probabilistic Early Expiration: For hot keys, randomly expire early to stagger refresh
- Local In-process Locks: Fast path for same-process requests
Invalidation Strategy:
- Write-invalidate: On write, invalidate all levels
- Async CDN Purge: CDN purges are async (can take seconds)
- TTL-based: CDN relies on HTTP cache-control headers
- Write-through option: For read-after-write consistency needs
TTL Hierarchy:
CDN (1h) → Edge (5min) → Service (1min) → DB (0)- Shorter TTLs closer to DB for freshness
- Longer TTLs at edge/CDN for performance
Cache Warming:
- Predictive warming based on access patterns
- Background refresh before TTL expiry
- Async population from higher cache levels
9. Monitoring & Metrics
class CacheMetrics: def __init__(self): self.hits = defaultdict(int) # level -> count self.misses = defaultdict(int) self.stampedes = 0 self.evictions = 0 def record(self, level: CacheLevel, hit: bool): if hit: self.hits[level] += 1 else: self.misses[level] += 110. Production Considerations
- Cold Start: On service restart, service cache empty → edge cache warms it
- Cache Penetration: Bloom filters for non-existent keys (cache negative results with short TTL)
- Cache Breakdown: Different keys sharded to different cache nodes
- Network Partitions: Fallback to lower cache levels if upper level unavailable
- Memory Pressure: LRU eviction at service/edge level
- Serialization: Use efficient binary format (msgpack, protobuf)
- Observability: Log cache hits/misses at each level, stampede events
11. HTTP Integration (CDN)
# For web applications, use HTTP cache headers def set_cache_headers(response, ttl_seconds: int): response.headers['Cache-Control'] = f'public, max-age={ttl_seconds}' response.headers['CDN-Cache-Control'] = f'max-age={ttl_seconds*6}' # CDN caches longer response.headers['Edge-Cache-Tag'] = f'key:{hash_key}'This architecture provides:
- ✅ Multi-level hierarchy with optimal TTLs
- ✅ Multiple stampede defenses (coalescing, locks, early expiration)
- ✅ Write invalidation with async CDN purge
- ✅ Predictive warming for hot keys
- ✅ Observability and metrics
- ✅ Production-ready with fallbacks and sharding
The key insight: Caches closer to user (CDN) are slower to update but faster to read; caches closer to DB (service) are faster to update but slower to read. TTLs and invalidation strategies must account for this tradeoff.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.