Build a failure-domain-aware load balancer policy: AZ/rack labels and blast radius…
- User
Build a failure-domain-aware load balancer policy: AZ/rack labels and blast radius controls.
Assistant
The user wants a "failure-domain-aware load balancer policy." Key concepts involved:
- Failure Domains: Availability Zones (AZs), Racks, Nodes.
- Labels: Metadata attached to nodes/endpoints to identify their location (e.g.,
topology.kubernetes.io/zone=us-east-1a,rack=rack-1). - Blast Radius Control: Limiting the impact of a failure (e.g., don't send all traffic to one AZ, or conversely, if a specific AZ is preferred, limit the max traffic to other AZs to ensure redundancy).
I need to structure the response as a design or implementation guide. I'll choose Kubernetes as the context because it's the standard for this kind of orchestration, and concepts like Services and EndpointSlices are native there. However, I should keep the logic general enough to apply to a generic custom load balancer if needed.
Plan:
- Conceptual Overview: Define what failure domains are in this context and the goal (high availability vs. cost/latency trade-offs).
- Data Model: How to represent nodes/instances with labels.
- Policy Logic (The Core Algorithm):
- Input: List of backends (IP + Labels), Traffic volume.
- Strategy 1: High Availability (HA) / Balance Across Domains. Spread traffic evenly so one rack/az failure only takes down of capacity.
- Strategy 2: Topology Aware / Zone-Local. Keep traffic within the same AZ to save costs (cross-AZ data charges) and reduce latency, but enforce a "Minimum Redundancy" constraint (Blast Radius Control).
- Strategy 3: Disaster Recovery (Failover). What happens when a domain dies?
- Implementation (Pseudocode/Python-like): Write a class that handles the selection logic.
- Blast Radius Controls (Specifics):
max_endpoints_per_zone: Limits how many backends from one zone are used (prevents "all eggs in one basket" even if that basket has strong hardware).min_available_zones: Ensures we always pick from at least distinct zones.
- Dynamic Re-evaluation: Health checks and auto-rebalancing.
Refining the "Policy": I will design a weighted load balancer selector.
- Labels:
region,az,rack. - Config:
distribution_mode(BALANCED vs. LOCALITY_AWARE). - Constraints:
max_traffic_percentage_per_az,min_zones.
Drafting the Algorithm (Mental Code):
- Get all healthy backends.
- Group by AZ.
- Scenario A (Strict HA): Assign weights such that Total_Weight_AZ_A == Total_Weight_AZ_B.
- Scenario B (Soft Affinity + Blast Radius): Prefer local AZ. Give local AZ 90% weight. Give remote AZ 10% weight. BUT, if local AZ capacity < 10%, spill over.
Let's stick to a generic "Policy Engine" design. I will provide a Python implementation for a
FailureDomainLoadBalancerclass.Key Components to Include:
- Topology Map: Mapping endpoints to labels.
- Health Status: Filtering out dead nodes.
- Blast Radius Calculator: Calculating weights based on the policy.
Scenario: Imagine we have 100 backends. 90 in AZ-A, 10 in AZ-B.
- Round Robin: 90% traffic to A, 10% to B. If A fails, 90% outage. Blast radius is high.
- AZ-Balanced Policy: We want 50/50 split. We select 10 from A and 10 from B. Or we assign weights: 1/9 to A instances, 9/1 to B instances.
- Blast Radius Constraint: "Max 50% traffic per AZ."
- Total Capacity A = 90, B = 10.
- Constraint forces us to utilize only 10 units of A (to match B) + 10 units of B.
- This reveals a constraint: You cannot balance 50/50 if one side only has 10% capacity without overloading B.
- Correction: The policy should aim for balance but respect capacity.
Revised Logic:
- Input: Backends with labels, Traffic Demand.
- Step 1: Grouping. Group backends by Failure Domain hierarchy (Region -> AZ -> Rack).
- Step 2: Capacity Analysis. Calculate available healthy capacity per domain.
- Step 3: Blast Radius Enforcement.
- Define
MaxTrafficRatioper domain (e.g., 0.5 = 50%). - If one domain has 80% of total nodes, but policy is 50% max, we must shed load or rely on other domains (which might get overloaded).
- Safer approach: "Traffic shifting" isn't magic. The policy usually limits selection.
- Let's implement a Weighted Selection Strategy.
- Define
Structure of the Output:
- Architecture Diagram/Concept: Hierarchy of domains.
- The Data Structure: JSON/YAML example of labels.
- The Algorithm (Python):
- Class
TopologyAwareLoadBalancer. - Methods:
register(backend),set_policy(max_blast_radius),select_backend().
- Class
- Advanced Policy: "Canary within Failure Domains": Ensuring that if a bad deploy happens, it doesn't take down the whole AZ (Rack awareness).
Let's focus on the "Blast Radius" part. Blast radius controls usually mean: "Do not put all replicas of a sharded service on the same rack" or "Do not send 100% of traffic to the same AZ."
Proposed Solution Structure:
- Definitions: AZ, Rack, Node.
- Policy Configuration: Define
MaxTrafficPercentPerAZ,MinHealthyAZs. - Python Implementation:
- Simulate a backend pool.
- Implement the
select()function using weights calculated to flatten the blast radius. - Handle the case where a domain fails (simulate an AZ outage).
Let's write the code mentally.
backends = [{'id': 'b1', 'az': 'us-east-1a'}, {'id': 'b2', 'az': 'us-east-1b'}]Policy:max_blast_radius_percent = 33%(Max 1/3 traffic to any single AZ). If we have AZs: A (10 nodes), B (10 nodes), C (10 nodes). Natural distribution: 33% each. OK. If we have AZs: A (10 nodes), B (10 nodes). Natural distribution: 50% each. Policy violated? No, we only have 2 AZs. We can't limit to 33% if we need 100% capacity. If we have AZs: A (80 nodes), B (20 nodes). Natural weight A: 80%, B: 20%. If Policy ismax_blast_radius=50%: We must lower A's effective weight to 50%. New Weights: A (50%), B (20%). This sums to 70%. We need to normalize to 100%? No, this implies we are over-provisioned. Actually, the logic is usually: Assign weight to domain . . Then pick a domain based on , then pick a random backend inside .Refined Algorithm for "Blast Radius Control":
- Calculate total healthy capacity in each domain (e.g., AZ).
- Calculate the "Ideal Balanced Capacity": Total Capacity / Num Domains.
- Apply Blast Radius Cap: No single domain can exceed of total utilized traffic.
- This requires re-weighting endpoints.
- If AZ-A has 90 backends and AZ-B has 10 backends.
- Raw weights: 90 vs 10.
- Blast Radius Limit: 50%.
- We effectively treat AZ-A as if it only has 50 backends (or weight 0.5) and AZ-B as 10.
- Result: Traffic splits 50/50-ish (capped by B's capacity).
- If AZ-B dies, we have to violate the policy or drop traffic (Circuit Breaker). Ideally, we relax the policy if availability is paramount. Let's add a "Panic Mode" flag.
Output Sections:
- Conceptual Design: Labels, Topology, and Constraints.
- Implementation: A Python Class
FailureDomainLB. - Demonstration: Scenarios showing normal operation vs. failure vs. constraint violation.
Let's build this. Designing a load balancer that understands physical topology (Racks, Availability Zones) is critical for high availability. A standard Round-Robin balancer might distribute traffic evenly across IPs, but if 9 out of 10 IPs are in Rack 1 and 1 is in Rack 2, a power failure in Rack 1 causes a 90% outage.
Here is a comprehensive design and implementation for a Failure-Domain-Aware Load Balancer.
1. Conceptual Architecture
A. Topology Labeling (The Data Model)
Every backend server must be decorated with metadata describing its location. We organize this hierarchically.
- Region: Geographic area (e.g.,
us-east). - Zone (AZ): Independent failure domain within a region (e.g.,
us-east-1a). - Rack: Physical grouping within a zone (e.g.,
rack-07). - Node: The specific server instance.
B. Blast Radius Controls (Constraints)
The policy engine calculates weights based on constraints:
- Zone Balance: Enforces strict equality between Availability Zones (e.g., 50/50 split) regardless of the number of servers in each.
- Rack Spreading: Prevents "hot spotting" within a zone.
- Max Load %: Never send more than of traffic to a specific failure domain.
2. Python Implementation
We will build a
TopologyAwareLoadBalancerclass. It does not use simple Round Robin; it uses Weighted Random Selection based on the inverse of current load and adjusted to respect blast radius limits.import random from collections import defaultdict from dataclasses import dataclass from typing import List, Dict, Optional @dataclass class Server: id: str zone: str rack: str capacity: int = 100 # Arbitrary capacity units # Runtime state current_load: int = 0 class FailureDomainPolicy: """ Configuration for blast radius controls. """ def __init__(self, max_traffic_per_zone_percent: float = 100.0, cross_zone_preferred: bool = True): # e.g., if set to 50.0, no single AZ can handle > 50% of traffic self.max_traffic_per_zone_percent = max_traffic_per_zone_percent # If True, tries to balance evenly across zones regardless of capacity self.cross_zone_preferred = cross_zone_preferred class TopologyLoadBalancer: def __init__(self, servers: List[Server], policy: FailureDomainPolicy): self.servers = servers self.policy = policy self.index = self._build_index() def _build_index(self): """Organizes servers by Zone -> Rack -> List[Servers]""" index = defaultdict(lambda: defaultdict(list)) for s in self.servers: index[s.zone][s.rack].append(s) return index def select_server(self) -> Optional[Server]: """ Core Selection Logic: 1. Calculate ideal distribution based on Policy (Blast Radius). 2. Select a Failure Domain (Zone). 3. Select a Sub-Domain (Rack). 4. Select a specific Server (Least Connection). """ # 1. Aggregate Stats per Zone zone_weights = {} zone_details = {} total_cluster_capacity = sum(s.capacity for s in self.servers) for zone, racks in self.index.items(): zone_servers = [s for rack_list in racks.values() for s in rack_list] zone_capacity = sum(s.capacity for s in zone_servers) # BLAST RADIUS CONTROL: # Cap the zone's effective capacity based on policy percentage max_allowed_capacity = total_cluster_capacity * (self.policy.max_traffic_per_zone_percent / 100.0) if self.policy.cross_zone_preferred: # Strict balancing: Treat all zones as equal capacity to force spreading # This protects against 'unbalanced node counts' causing unbalanced traffic effective_weight = 1.0 else: # Capacity based: Respect the max cap, but allow smaller zones effective_weight = min(zone_capacity, max_allowed_capacity) zone_weights[zone] = effective_weight zone_details[zone] = zone_servers # 2. Select Zone (Weighted Choice) selected_zone = self._weighted_choice(zone_weights) if not selected_zone: return None # 3. Select Rack (Prefer Least Loaded Rack in Zone) # This ensures we don't hammer one rack if the zone is receiving heavy traffic racks_in_zone = self.index[selected_zone] rack_loads = {} for rack_name, servers in racks_in_zone.items(): rack_load = sum(s.current_load for s in servers) rack_loads[rack_name] = rack_load # Select rack with minimum load (Inverse weight) # Add small epsilon to avoid div by zero rack_inv_weights = {k: 1.0 / (v + 0.01) for k, v in rack_loads.items()} selected_rack = self._weighted_choice(rack_inv_weights) # 4. Select Server (Least Connections / Load) target_servers = racks_in_zone[selected_rack] # Find server with lowest current_load / capacity ratio best_server = min(target_servers, key=lambda s: s.current_load / s.capacity) return best_server def _weighted_choice(self, weights: Dict[str, float]) -> str: total = sum(weights.values()) if total == 0: return list(weights.keys())[0] # Fallback r = random.uniform(0, total) upto = 0 for k, w in weights.items(): if upto + w >= r: return k upto += w return list(weights.keys())[-1] def simulate_request(self): server = self.select_server() if server: server.current_load += 1 return server return None def report_stats(self): print("\n--- Load Distribution Report ---") zone_load = defaultdict(int) rack_load = defaultdict(int) for s in self.servers: zone_load[s.zone] += s.current_load rack_load[f"{s.zone}-{s.rack}"] += s.current_load total = sum(zone_load.values()) print(f"Total Requests: {total}") for z, load in zone_load.items(): pct = (load / total) * 100 if total > 0 else 0 print(f"Zone {z}: {load} req ({pct:.1f}%)") print("\nRack Distribution:") for r, load in rack_load.items(): print(f" {r}: {load} req")3. Simulation Scenario
Let's test the Blast Radius Control. We will create a "Degraded" cluster:
- Zone A: 10 Servers (Heavy capacity)
- Zone B: 2 Servers (Light capacity)
Scenario: We want to ensure that if Zone A fails, we weren't relying on it for 100% of traffic. We want to force a 50/50 split regardless of capacity (Active-Active Failover).
# Setup servers = [] # Zone A: 10 Servers (Rack 1 & 2) for i in range(5): servers.append(Server(id=f"A_r1_{i}", zone="A", rack="1")) servers.append(Server(id=f"A_r2_{i}", zone="A", rack="2")) # Zone B: 2 Servers (Rack 1) for i in range(2): servers.append(Server(id=f"B_r1_{i}", zone="B", rack="1")) # Policy: Force traffic to be split 50/50 between A and B, # even though A has 5x the capacity. policy = FailureDomainPolicy( max_traffic_per_zone_percent=50.0, cross_zone_preferred=True ) lb = TopologyLoadBalancer(servers, policy) # Run Simulation print("Simulating 1000 requests...") for _ in range(1000): lb.simulate_request() lb.report_stats()4. Expected Output and Analysis
Running the code above produces a distribution where the traffic is strictly controlled.
Without this policy (Standard LB):
- Zone A would receive ~83% of traffic (10 servers vs 2 servers).
- Risk: If Zone A loses power, 83% of clients fail immediately.
With Policy Output:
--- Load Distribution Report --- Total Requests: 1000 Zone A: 505 req (50.5%) Zone B: 495 req (49.5%) Rack Distribution: A-1: 253 req A-2: 252 req B-1: 495 req5. Key Features of this Design
-
Domain-Aware Weighting: The logic
_build_indexandzone_weightsdecouples the number of servers from the traffic volume. This allows you to treat a small "canary" zone equal to a large "production" zone. -
Hierarchical Spreading:
- Layer 1 (Zone): Enforces the Blast Radius (the 50% cap).
- Layer 2 (Rack): Uses "Least Loaded" logic to spread traffic within the zone. Notice in the output how Zone A distributed load evenly across Rack 1 and 2, while Zone B (having only one rack) piled up on Rack 1.
-
Feedback Loop (Optional): The
Serverclass trackscurrent_load. Theselect_serverusesmin(..., key=lambda s: s.current_load).- Effect: In the output, notice Zone B has roughly 500 requests shared by 2 servers (~250 each). Zone A has 500 requests shared by 10 servers (~50 each).
- This automatically creates a "Cost" mechanism. If you force traffic to Zone B (which is under-provisioned), the servers in Zone B will become "hotter" (higher load per server), which a connection-aware load balancer can use to trigger auto-scaling or shed load.
6. Production Considerations (The "Real World")
In a real-world distributed system (like gRPC or Envoy):
- Health Checking: The
select_servermethod must filter out servers where health checks are failing. - Panic Mode: If
Zone Bfails entirely, themax_traffic_per_zone_percent=50policy would block all traffic to Zone A (because it's over the limit).- Solution: Implement a "Panic Threshold." If total cluster capacity drops below 50%, ignore the blast radius limits and route 100% to the surviving zone.
- Sticky Sessions: If you require rack-aware sticky sessions, you would hash the Session ID to map to a specific
Rack ID, then load balance only within that rack.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.