Design a shadow traffic diff analyzer: schema-aware comparisons, tolerance rules, and…
- User
Design a shadow traffic diff analyzer: schema-aware comparisons, tolerance rules, and alerting.
- Assistant
We are going to design a shadow traffic diff analyzer that compares shadow traffic (replayed requests) against the primary traffic (live requests) in a schema-aware manner, with tolerance rules and alerting capabilities.
Key Components:
-
Schema Awareness:
- We need to know the structure of the data (requests and responses) to compare them meaningfully. This could be achieved by having a schema definition (e.g., OpenAPI for REST, Protobuf for gRPC) or by dynamically inferring schema from the traffic.
- The system should be able to compare fields at the appropriate level (e.g., compare numbers with tolerance for floating point, ignore certain fields, etc.).
-
Tolerance Rules:
- Some fields might be allowed to have small differences (e.g., timestamps, IDs, or floating-point calculations with rounding errors).
- We should allow users to define tolerance rules per field or per type (e.g., all timestamps within 1 second, all doubles within 0.001).
- Also, we might want to ignore certain fields entirely (e.g., nonce, request ID, etc.).
-
Alerting:
- When differences exceed the defined tolerances, we want to generate alerts.
- Alerts should be configurable (e.g., threshold for the number of differences, etc.) and should include enough context (request/response details, difference description, etc.).
-
Comparison Process:
- We will have two streams of traffic: primary (live) and shadow (replayed). They might not be aligned one-to-one, so we might need to match requests (e.g., by some correlation ID or by matching the request parameters).
- We compare the matched pairs (or if we cannot match, we might note that as a difference).
-
Output:
- A report of differences, possibly with statistics (e.g., percentage of requests with differences, types of differences, etc.).
- Alerts when the differences exceed a threshold.
Design:
Let's break down the design into modules:
1. Schema Management Module:
- Stores schemas (from files or inferred from traffic).
- Provides a way to look up the schema for a given endpoint or request type.
- Might also store tolerance rules per field.
2. Traffic Ingestion and Matching:
- Ingests two streams: primary and shadow.
- Tries to match requests from primary to shadow. We can use a few strategies: a. By a correlation ID if present (like a trace ID). b. By matching key fields (like user ID, product ID, etc.) with some fuzzy matching. c. If no match, then we might have a mismatch (shadow request without primary or vice versa).
- We might also consider temporal matching (within a time window) if the traffic is continuous.
3. Comparison Engine:
- For each matched pair (primary request/response and shadow request/response): a. Compare the requests: are they the same? (considering tolerances and ignored fields) b. Compare the responses: are they the same? (with tolerances)
- The comparison should be recursive for nested objects (like JSON). We need to handle:
- Primitives: numbers, strings, booleans, null.
- Arrays and objects.
4. Tolerance Rule Engine:
- Applies the tolerance rules during comparison. For example:
- If the field is a number and has a tolerance defined (absolute or relative), then check if the difference is within tolerance.
- If the field is a string and has an ignore rule, skip comparison.
- If the field is a timestamp, convert to a standard format and compare with tolerance.
5. Difference Recording:
- Record each difference with:
- The matched pair (or unmatched request)
- The field path (e.g., "response.body.user.age")
- The expected value (from primary) and actual value (from shadow)
- The type of difference (mismatch, missing field, extra field, etc.)
6. Alerting Module:
- Based on the differences recorded, check if any alert condition is met.
- Conditions could be: a. Total number of mismatches exceeds a threshold. b. Mismatch rate (over a time window or total) exceeds a threshold. c. Specific fields (or endpoints) have too many mismatches.
- When an alert condition is met, send an alert (via email, Slack, etc.) with summary and possibly the first few differences.
7. Reporting and Dashboard:
- Provide a way to view the differences, statistics, and history.
Implementation Considerations:
- We might want to run this in near real-time (as traffic comes) or in batch (after collecting a set of traffic).
- For performance, we might need to sample the traffic if the volume is high.
Let's sketch the classes/interfaces:
Schema Management:
class SchemaManager: def get_schema(endpoint: str, method: str) -> Schema: # returns a Schema object that describes the structure of the request and response. class Schema: # Could be a generic tree structure with nodes for each field, having: # - name # - type (primitive, array, object) # - constraints (min, max, pattern, etc.) # - tolerance_rules (optional) # - ignore (boolean)Tolerance Rules:
We can define a ToleranceRule per field (or per type). A rule could be:
class ToleranceRule: def is_tolerated(expected, actual) -> bool: # returns True if the difference is within tolerance.Examples:
- AbsoluteToleranceRule(delta=0.001) for floats.
- RelativeToleranceRule(ratio=0.01) for floats.
- TimeToleranceRule(seconds=1) for timestamps.
- IgnoreRule() for strings that we want to ignore.
We can also have a CompositeToleranceRule that combines multiple rules.
Traffic Matching:
We need a way to match requests. Let's assume we have:
class TrafficPair: primary_request: Request primary_response: Response shadow_request: Request shadow_response: Response class Matcher: def match(primary_stream, shadow_stream) -> List[TrafficPair]: # returns a list of matched pairs. Also, we might return unmatched requests.The matching strategy can be pluggable.
Comparison:
class Difference: field_path: str # e.g., "body.user.age" expected: Any actual: Any difference_type: str # "mismatch", "missing_in_shadow", "missing_in_primary", etc. class Comparator: def compare(primary: Request, shadow: Request, schema: Schema, tolerance_rules: Dict[str, ToleranceRule]) -> List[Difference]: # compare the two objects (could be request or response) according to the schema and tolerance rules. # returns a list of differences.Alerting:
class AlertCondition: # could be a function that takes the current state (number of differences, etc.) and returns True if alert should be raised. # or we can have a set of rules like: # - total_differences > 100 # - mismatch_rate > 0.1 (over the last 1000 requests) # - specific field X has more than 10 mismatches. class AlertEngine: def __init__(self, conditions: List[AlertCondition]): self.conditions = conditions def evaluate(self, differences: List[Difference], total_requests: int) -> List[Alert]: # returns a list of alerts that are triggered.Overall Flow:
- Ingest primary and shadow traffic (maybe from logs or real-time streams).
- For each incoming request (or batch), try to match with the other stream.
- For each matched pair, look up the schema (by endpoint and method) and the tolerance rules (by field paths).
- Compare the requests and then the responses, collecting differences.
- Feed the differences to the alert engine and also store for reporting.
- Periodically (or after each batch) evaluate alert conditions and send alerts if needed.
Challenges:
-
Matching without a correlation ID is hard. We might need to define matching keys (like a combination of fields) and allow some tolerance in matching too (e.g., timestamps within a few seconds). But note: if we match incorrectly, we might get false differences.
-
Schema evolution: the schema might change over time. We need to handle the schema versioning or dynamic schema lookup based on the request.
-
Performance: if we have high traffic, we might need to sample. Also, the comparison could be expensive for large payloads.
-
Statefulness: matching might require state (to hold requests for a time window until a match is found). We need to manage this state and clean up old requests.
Example:
Suppose we have an endpoint
GET /users/{id}. The schema says the response has a fieldcreated_atwhich is a timestamp. We set a tolerance of 1 second for that field. Then, when comparing, if the primary response hascreated_at: 2024-01-01T12:00:00Zand shadow has2024-01-01T12:00:01Z, that's tolerated.Another example: the request has a field
noncewhich is random. We set an ignore rule for that field.Alert Example:
We set an alert condition: if the number of mismatches in the last 1000 requests is more than 5, then alert.
Extensions:
-
We might want to compare the entire request/response as a string (with some normalization) if we don't have a schema. But schema-aware is more powerful.
-
We might want to compare the performance (latency) as well? That could be another metric.
Given the complexity, we might start with a simpler version that assumes we have a correlation ID and a fixed schema, and then extend.
Let's write a high-level code structure for the main components:
class ShadowTrafficAnalyzer: def __init__(self, schema_manager, matcher, comparator, alert_engine, reporter): self.schema_manager = schema_manager self.matcher = matcher self.comparator = comparator self.alert_engine = alert_engine self.reporter = reporter def process_batch(self, primary_requests, shadow_requests): # Step 1: Match pairs, unmatched_primary, unmatched_shadow = self.matcher.match(primary_requests, shadow_requests) # Step 2: For each pair, compare all_differences = [] for pair in pairs: # Get schema for the endpoint and method (from primary or shadow? they should be the same) schema = self.schema_manager.get_schema(pair.primary_request.endpoint, pair.primary_request.method) # Get tolerance rules for the fields in the schema (could be stored in the schema or separate) tolerance_rules = self._get_tolerance_rules(schema) # by field path # Compare requests req_differences = self.comparator.compare( pair.primary_request.body, pair.shadow_request.body, schema.request_schema, tolerance_rules ) # Adjust field path prefix for request differences? e.g., "request." + field_path req_differences = [self._prefix_field_path(d, "request") for d in req_differences] # Compare responses resp_differences = self.comparator.compare( pair.primary_response.body, pair.shadow_response.body, schema.response_schema, tolerance_rules ) resp_differences = [self._prefix_field_path(d, "response") for d in resp_differences] all_differences.extend(req_differences + resp_differences) # Also record unmatched as differences? Maybe as "missing" in one side. for req in unmatched_primary: all_differences.append(Difference( field_path=" Entire request", expected="Primary request present", actual="Shadow request missing", difference_type="missing_in_shadow" )) for req in unmatched_shadow: all_differences.append(Difference( field_path=" Entire request", expected="Shadow request present", actual="Primary request missing", difference_type="missing_in_primary" )) # Step 3: Alerting alerts = self.alert_engine.evaluate(all_differences, len(primary_requests) + len(shadow_requests)) # Step 4: Reporting self.reporter.report(all_differences, alerts) return alertsThis is a simplified version. In reality, we might process requests one by one (streaming) and maintain state for matching (like a window of recent requests). Also, the tolerance rules might be stored in the schema or in a separate configuration.
Tolerance Rule Lookup:
We can have a way to get the tolerance rule for a given field path. The field path is a dot-separated string (like "body.user.age"). We can have:
class ToleranceRuleProvider: def get_rule(field_path: str) -> Optional[ToleranceRule]: # returns the rule for that field, or None if no rule (meaning exact match).The rules could be defined in a configuration file or in the schema.
Matcher:
The matcher needs to be stateful if we are processing streams. We might keep two queues (for primary and shadow) and try to match within a time window.
A simple matcher with correlation ID:
class CorrelationIdMatcher: def __init__(self, correlation_id_field="trace_id"): self.correlation_id_field = correlation_id_field self.primary_requests = [] # could be a dict by correlation_id self.shadow_requests = [] # same def match(self, primary_requests, shadow_requests): # add the new requests to the respective lists (or dicts by correlation_id) # then try to match by correlation_id. # also, remove old requests (older than the window) from the lists.But note: the correlation ID might be in the request body or header. We need to extract it.
Alternatively, if we don't have a correlation ID, we might try to match by some business key (like user_id, product_id, etc.) and also within a time window. This is more complex and might lead to false matches.
Given the complexity of matching without a correlation ID, it might be better to require the primary and shadow traffic to have a common identifier. If not, we might have to resort to heuristics.
Conclusion:
We have outlined a design for a shadow traffic diff analyzer that is schema-aware, supports tolerance rules, and has alerting. The key challenges are in matching and in defining the schema and tolerance rules. We can start with a simple implementation that assumes a correlation ID and fixed schemas, then extend to more dynamic scenarios. I'll design a comprehensive shadow traffic diff analyzer with schema-awareness, tolerance rules, and intelligent alerting.
Architecture Overview
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Traffic │ │ Schema │ │ Tolerance │ │ Ingestion │───▶│ Registry │───▶│ Engine │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Matcher │ │ Diff │ │ Alert │ │ (Correlation) │───▶│ Calculator │───▶│ Engine │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ │ ▼ ▼ ┌─────────────────┐ ┌──────────────────┐ │ State Store │ │ Dashboard/ │ │ (Redis/DB) │ │ Reporting │ └─────────────────┘ └──────────────────┘Core Components
1. Schema-Aware Comparator
class SchemaAwareComparator: def __init__(self, schema_registry: SchemaRegistry): self.schema_registry = schema_registry def compare( self, primary: RequestResponse, shadow: RequestResponse, endpoint: str, method: str ) -> ComparisonResult: """ Schema-aware comparison with nested object traversal """ schema = self.schema_registry.get_schema(endpoint, method) differences = [] # Compare request structure first req_diff = self._compare_objects( primary.request.body, shadow.request.body, schema.request_schema, path="request" ) differences.extend(req_diff) # Compare response with response schema resp_diff = self._compare_objects( primary.response.body, shadow.response.body, schema.response_schema, path="response" ) differences.extend(resp_diff) # Compare metadata (headers, status, latency) meta_diff = self._compare_metadata(primary, shadow) differences.extend(meta_diff) return ComparisonResult( match=len(differences) == 0, differences=differences, primary_request_id=primary.request.id, shadow_request_id=shadow.request.id )2. Tolerance Rule Engine
class ToleranceRuleEngine: def __init__(self): self.rules: Dict[str, List[ToleranceRule]] = {} def add_rule( self, endpoint_pattern: str, field_path: str, rule: ToleranceRule ): """Add tolerance rule for specific field in endpoint pattern""" key = f"{endpoint_pattern}:{field_path}" if key not in self.rules: self.rules[key] = [] self.rules[key].append(rule) def is_tolerated( self, endpoint: str, field_path: str, expected: Any, actual: Any ) -> Tuple[bool, Optional[str]]: """ Check if difference is within tolerance. Returns (is_tolerated, reason_if_not) """ # Find matching rules rules = self._get_matching_rules(endpoint, field_path) for rule in rules: if rule.matches(expected, actual): return True, rule.description return False, None def _get_matching_rules(self, endpoint: str, field_path: str) -> List[ToleranceRule]: """Get all rules matching endpoint pattern and field""" matched = [] for pattern, rules in self.rules.items(): ep_pattern, field_pattern = pattern.split(':', 1) if fnmatch.fnmatch(endpoint, ep_pattern) and \ fnmatch.fnmatch(field_path, field_pattern): matched.extend(rules) return matchedTolerance Rule Types:
class ToleranceRule(ABC): @abstractmethod def matches(self, expected: Any, actual: Any) -> bool: pass @property @abstractmethod def description(self) -> str: pass class AbsoluteNumericTolerance(ToleranceRule): def __init__(self, delta: float): self.delta = delta def matches(self, expected: Any, actual: Any) -> bool: if not isinstance(expected, (int, float)) or \ not isinstance(actual, (int, float)): return False return abs(expected - actual) <= self.delta class PercentageTolerance(ToleranceRule): def __init__(self, percent: float): self.percent = percent / 100.0 def matches(self, expected: Any, actual: Any) -> bool: if expected == 0: return actual == 0 if not isinstance(expected, (int, float)): return False return abs((actual - expected) / expected) <= self.percent class TimestampTolerance(ToleranceRule): def __init__(self, seconds: int, unit: str = 's'): self.seconds = seconds def matches(self, expected: Any, actual: Any) -> bool: try: exp_dt = parse_timestamp(expected) act_dt = parse_timestamp(actual) return abs((act_dt - exp_dt).total_seconds()) <= self.seconds except: return False class EnumValueTolerance(ToleranceRule): def __init__(self, allowed_values: Set[Any]): self.allowed = allowed_values def matches(self, expected: Any, actual: Any) -> bool: return actual in self.allowed class RegexTolerance(ToleranceRule): def __init__(self, pattern: str): self.regex = re.compile(pattern) def matches(self, expected: Any, actual: Any) -> bool: if not isinstance(actual, str): return False return bool(self.regex.match(actual))3. Intelligent Matcher
class TrafficMatcher: def __init__(self, correlation_config: CorrelationConfig): self.correlation_config = correlation_config self.pending_primary = SortedDict(key=lambda x: x.timestamp) self.pending_shadow = SortedDict(key=lambda x: x.timestamp) def match( self, primary_traffic: List[RequestResponse], shadow_traffic: List[RequestResponse] ) -> List[MatchedPair]: """ Match primary and shadow traffic using multiple strategies """ matched_pairs = [] # Strategy 1: Correlation ID (highest priority) if self.correlation_config.use_correlation_id: matched_pairs.extend(self._match_by_correlation_id( primary_traffic, shadow_traffic )) # Strategy 2: Business key matching if self.correlation_config.use_business_key: matched_pairs.extend(self._match_by_business_key( primary_traffic, shadow_traffic, self.correlation_config.business_keys )) # Strategy 3: Temporal + fuzzy matching matched_pairs.extend(self._match_by_temporal_fuzzy( primary_traffic, shadow_traffic, window_ms=self.correlation_config.time_window_ms, similarity_threshold=self.correlation_config.similarity_threshold )) return matched_pairs4. Alert Engine
class AlertEngine: def __init__( self, thresholds: AlertThresholds, alert_channels: List[AlertChannel] ): self.thresholds = thresholds self.channels = alert_channels self.metrics_store = MetricsStore() def evaluate( self, comparison_results: List[ComparisonResult], window: TimeWindow ) -> List[Alert]: """ Evaluate alerts based on multiple metrics """ alerts = [] # Calculate metrics metrics = self._calculate_metrics(comparison_results, window) # Check threshold violations violations = self._check_thresholds(metrics) for violation in violations: alert = Alert( level=violation.level, title=violation.title, description=violation.description, metrics=metrics, timestamp=datetime.utcnow(), context=self._gather_context(violation, comparison_results) ) alerts.append(alert) # Send via configured channels for channel in self.channels: channel.send(alert) return alerts def _calculate_metrics( self, results: List[ComparisonResult], window: TimeWindow ) -> TrafficMetrics: """Calculate various metrics from comparison results""" total = len(results) mismatches = sum(1 for r in results if not r.match) # Group by endpoint by_endpoint = defaultdict(lambda: {'total': 0, 'mismatches': 0}) for result in results: key = result.endpoint by_endpoint[key]['total'] += 1 if not result.match: by_endpoint[key]['mismatches'] += 1 # Field-level analysis field_stats = self._analyze_field_differences(results) return TrafficMetrics( total_requests=total, mismatch_count=mismatches, mismatch_rate=mismatches / total if total > 0 else 0, by_endpoint=dict(by_endpoint), field_difference_stats=field_stats, window=window )Alert Thresholds Configuration:
thresholds: global: mismatch_rate: warning: 0.01 # 1% critical: 0.05 # 5% endpoint_specific: "/api/v1/payment/*": mismatch_rate: critical: 0.0 # No tolerance for payment endpoints field_whitelist: - "response.headers.x-request-id" # Ignore differences in these fields "/api/v1/products/*": field_blacklist: - "response.body.timestamp" # Always ignore timestamp differences anomaly_detection: enabled: true sensitivity: 2.0 # Standard deviations for anomaly detection min_samples: 100 # Minimum samples for statistical significance5. Dashboard & Reporting
class Dashboard: def __init__(self, db: Database): self.db = db def get_summary( self, time_range: TimeRange, filters: Dict[str, Any] = None ) -> DashboardSummary: """ Generate summary dashboard with: - Mismatch rate over time - Top mismatched endpoints - Top mismatched fields - Recent alerts - Trend analysis """ pass def get_comparison_details( self, primary_request_id: str, shadow_request_id: str ) -> DetailedComparison: """ Show side-by-side comparison with: - Schema validation - Tolerance rule application - Highlighted differences - Raw payloads """ passConfiguration Example
# config/shadow-analyzer.yaml matcher: correlation_strategies: - type: "header" name: "X-Request-ID" required: true - type: "business_key" keys: ["user_id", "session_id", "order_id"] max_time_drift_ms: 5000 time_window_ms: 10000 fuzzy_match_threshold: 0.85 tolerances: - endpoint: "/api/v1/users/*" field: "response.body.last_login" rule: "timestamp" config: max_difference_seconds: 300 - endpoint: "/api/v1/orders/*" field: "response.body.total" rule: "percentage" config: tolerance_percent: 0.01 # 1% tolerance for price calculations - endpoint: "/*" field: "response.headers.date" rule: "ignore" - endpoint: "/api/v1/status" field: "response.body.server_time" rule: "regex" config: pattern: "\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}Z" alerting: thresholds: mismatch_rate: global: warning: 0.01 critical: 0.05 by_endpoint: "/api/v1/payment/submit": critical: 0.0 latency_increase: threshold_percent: 20 window_minutes: 5 channels: - type: "slack" webhook: "${SLACK_WEBHOOK}" channel: "#shadow-alerts" - type: "pagerduty" service_key: "${PD_SERVICE_KEY}" severity_mapping: warning: "warning" critical: "critical" suppression: group_by: ["endpoint", "field_path"] window_minutes: 15 max_alerts_per_window: 10 storage: comparison_results_ttl_days: 30 metrics_retention_days: 90 detailed_samples_keep: 1000 # Keep full details for N mismatches per window schema_registry: sources: - type: "openapi" url: "http://schema-registry/api-docs" refresh_interval_hours: 6 - type: "protobuf" descriptor_set: "/schemas/proto/descriptor.pb" fallback: infer_from_samples: true min_samples_for_inference: 100Deployment & Operations
1. High Availability Setup
Load Balancer ├── Analyzer Instance 1 (active) ├── Analyzer Instance 2 (standby) └── Analyzer Instance 3 (standby) Shared State: ├── Redis Cluster (for pending matches) ├── PostgreSQL (for stored results) └── S3/Blob Storage (for large payloads)2. Scaling Considerations
- Horizontal scaling: Stateless comparison workers behind a queue
- Sharding: By endpoint or tenant for multi-tenant deployments
- Backpressure: Drop samples if processing lags (configurable)
3. Monitoring
# Key metrics to monitor shadow_analyzer_requests_total shadow_analyzer_mismatches_total{endpoint, field} shadow_analyzer_match_latency_seconds shadow_analyzer_alerts_sent_total{channel, level} shadow_analyzer_tolerance_hits_total{rule_type} shadow_analyzer_schema_errors_total shadow_analyzer_queue_depth4. Sample Alert
{ "level": "critical", "title": "High mismatch rate on /api/v1/payment/submit", "description": "Mismatch rate 0.0% (expected ≤0.0%) in last 5 minutes. 3/3 requests mismatched.", "timestamp": "2024-01-15T10:30:00Z", "metrics": { "total_requests": 150, "mismatch_count": 8, "mismatch_rate": 0.0533, "by_endpoint": { "/api/v1/payment/submit": { "total": 3, "mismatches": 3, "mismatch_rate": 1.0 } }, "top_mismatched_fields": [ {"field": "response.body.amount", "count": 3}, {"field": "response.body.currency", "count": 2} ] }, "context": { "sample_mismatches": [ { "primary_request_id": "req_abc123", "shadow_request_id": "req_def456", "differences": [ { "path": "response.body.amount", "expected": 100.00, "actual": 100.01, "tolerated": false, "rule_applied": null } ] } ] } }Implementation Roadmap
Phase 1: Basic Schema-Aware Comparison
- Schema registry integration (OpenAPI/Protobuf)
- Basic structural comparison
- Simple numeric tolerance
Phase 2: Advanced Matching & Tolerance
- Correlation ID matching
- Business key matching
- Comprehensive tolerance rules
- Field-level ignore patterns
Phase 3: Alerting & Dashboard
- Multi-threshold alerting
- Dashboard with drill-down
- Historical analysis
Phase 4: Advanced Features
- Machine learning anomaly detection
- Automated tolerance suggestion
- Canary analysis integration
- Performance impact correlation
Key Design Decisions
- Schema-First Approach: Ensures comparisons are semantically meaningful
- Pluggable Matcher: Supports multiple correlation strategies
- Composable Tolerances: Rules can be combined and prioritized
- Contextual Alerts: Include sample mismatches for debugging
- Observability First: Extensive metrics and structured logging
- Graceful Degradation: Continue analysis if schema missing (fallback to structural comparison)
This design provides a production-ready shadow traffic analyzer that can detect subtle behavioral changes while accounting for acceptable variations in numeric calculations, timestamps, and non-deterministic fields.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.