Engineer a multi-tenant observability cost optimizer that suggests sampling changes,…
- User
Engineer a multi-tenant observability cost optimizer that suggests sampling changes, cardinality reductions, and retention adjustments.
- Assistant
We are building a multi-tenant observability cost optimizer that suggests changes to sampling, cardinality, and retention. The system will have to consider multiple tenants (users/organizations) and their specific needs and constraints. We need to design a solution that can analyze current usage and costs, and then provide actionable suggestions.
Steps:
-
Data Collection: We need to collect metrics about each tenant's current observability usage. This includes:
- Volume of ingested data (bytes/events per time period)
- Cardinality of high-cardinality dimensions (for metrics, logs, traces)
- Current sampling rates (if any)
- Retention policies (how long data is stored)
- Cost breakdown (if available) or at least the pricing model of the observability platform.
-
Cost Modeling: We need to model the cost for each tenant. The cost typically depends on:
- Ingested data volume (often per GB)
- High cardinality (often charged per cardinality point or by the number of active series)
- Retention (longer retention means more data stored, and storage cost)
- Possibly query costs (but we are focusing on ingestion and storage for now)
-
Analysis: For each tenant, we analyze:
- Is the tenant over-sampling? (e.g., sampling rate too high for the use case)
- Are there high-cardinality dimensions that could be reduced without losing critical insights?
- Is the retention period too long for the tenant's actual needs?
-
Suggestion Generation: Based on the analysis, generate specific, actionable suggestions. Example: "Reduce sampling rate for service X from 100% to 10% to save Z per month." Example: "Change retention from 30 days to 7 days for logs, saving $W per month."
-
Multi-Tenant Considerations:
- Each tenant might have different SLAs or requirements. We must not suggest changes that break their requirements.
- We need to know the tenant's tolerance for data loss or reduced granularity? Maybe we have a way to input constraints.
-
Implementation:
We can break down the system into:
a. Data Collector: Pulls data from the observability platform (like Prometheus, Loki, Jaeger, etc.) or from a cost reporting system. b. Cost Modeler: Takes the collected data and the pricing model and computes current cost and potential cost for different scenarios. c. Analyzer: Applies rules (or ML models) to identify optimization opportunities. d. Suggestion Generator: Produces human-readable suggestions with estimated savings and impact.
-
Constraints and Rules:
We must have a way to define what is acceptable for each tenant. For example:
- Some tenants might require 100% sampling for certain critical services.
- Some might have compliance requirements for retention.
We can have a configuration per tenant that sets:
- Minimum sampling rates per service or data type.
- Maximum allowed cardinality reduction (e.g., we cannot remove certain tags).
- Minimum retention period.
-
Example Workflow:
For a given tenant:
- Collect current configuration and usage.
- Compare against constraints (if any).
- For each service/data type, consider: a. Sampling: Can we reduce? (But not below min) -> calculate new volume and cost. b. Cardinality: Identify high-cardinality tags that are not required (maybe by looking at which tags are used in queries?). c. Retention: Can we reduce? (But not below min) -> calculate storage cost.
We might run simulations for each possible change and then pick the ones with highest savings that respect constraints.
-
Challenges:
- Cardinality reduction might break existing dashboards/alerts. We need to know which tags are used in queries? Maybe we can analyze query patterns.
- Sampling might affect the accuracy of SLO calculations. We need to know the tenant's SLOs? Maybe we have to avoid changing sampling for services that are used for SLOs.
-
Proposed Solution:
We can design a modular system:
Module 1: Tenant Configuration
- Stores per-tenant constraints and requirements.
Module 2: Usage Ingestion
- Periodically (or on-demand) fetch:
- Current sampling rates (by service, by data type)
- Current cardinality (number of active series for each metric, or distinct values for log labels, etc.)
- Current retention settings
- Query logs (to see which labels are used in queries) [optional but helpful]
Module 3: Cost Simulator
- Given a set of parameters (sampling, cardinality, retention) and the pricing model, compute cost.
- We can run multiple simulations (e.g., try reducing sampling by 10%, 20%, ... until we hit the constraint).
Module 4: Opportunity Detector
- Runs heuristics or rules to find potential changes:
- If a service has high cardinality and we see that some tags are never used in queries (over a period), suggest removing them.
- If a service has 100% sampling and the volume is very high, suggest reducing (if allowed) and check the impact on SLOs (if we have SLO definitions).
- If retention is set to 90 days but the tenant only queries last 7 days, suggest reducing.
Module 5: Suggestion Generator
- Takes the opportunities and formats them into a report for the tenant.
-
Example Heuristics:
a. Cardinality:
- For each metric, look at the labels. If a label has more than N distinct values (e.g., 100) and we see that in the last 30 days, it was never used in a query (from query logs), then we can suggest removing that label.
- But we must check if that label is used in alerts? We might not have that data. So we can be conservative and only suggest if we have query logs and the label is not used.
b. Sampling:
- For services that are not critical (maybe defined by the tenant) and have high traffic, we can suggest reducing sampling. We can also look at the current error rate and see if we are already sampling enough to detect errors.
c. Retention:
- Look at the tenant's query patterns. If 95% of queries are for the last 7 days, then we can suggest setting retention to 7 days for that data type.
-
Implementation Plan:
We can build this as a service that runs periodically (daily) and produces a report.
Steps:
-
For each tenant: a. Load tenant configuration (constraints, critical services, etc.) b. Fetch current usage and configuration from the observability platform. c. Fetch query logs (if available) for the last 30 days.
-
Run the opportunity detector:
For each service (or data source) and for each data type (metrics, logs, traces):
a. Sampling: - If current sampling is above the minimum allowed and the service is not critical, and the volume is high, then consider reducing. - Calculate the new volume and cost saving. Also, consider the impact on SLOs (if we have SLO definitions, we might need to check if the reduced sampling will still meet the SLO). This might be complex and require statistical analysis.
b. Cardinality: - For each metric (or log stream), list the labels. - For each label, count distinct values and check if the label is used in any query in the last 30 days (from query logs). - If a label is not used and the distinct value count is above a threshold (e.g., 10) and the label is not protected (in constraints), then we can suggest removing it. - Calculate the reduction in cardinality and the cost saving.
c. Retention: - Analyze query logs to see the time range of queries. For example, for logs, what is the maximum time range looked up? (e.g., 99% of queries look at data from the last 7 days) - If current retention is longer than the maximum time looked up (and above the minimum allowed), then suggest reducing retention to that maximum time (or a bit more, e.g., 1.5x) to allow for occasional longer queries.
-
For each opportunity, we must also consider the risk of breaking existing dashboards/alerts that we cannot see. So we might want to be conservative and only suggest changes that are safe (like removing unused labels) or that the tenant can test in a non-production environment.
-
Generate a report with:
- Current cost (if we can compute it)
- For each suggestion: what to change, estimated savings, and a note about the risk.
-
-
Tech Stack:
We can build this in Python (for data analysis) and use a database to store tenant configurations and historical suggestions.
We need to integrate with:
- Observability platform APIs (Prometheus, Loki, etc.) to get current config and usage.
- Cost data (maybe from the cloud provider or the observability vendor's billing API).
For query logs, if the platform does not provide, we might not be able to do the cardinality and retention analysis based on usage.
-
Limitations:
Without query logs, we cannot know which labels are used or what time ranges are queried. In that case, we might rely on heuristics that are less accurate.
We might also need to consider the tenant's business logic. For example, a tenant might be using a high-cardinality label for debugging even if it's not in regular queries.
-
Advanced:
We could use machine learning to predict the impact of sampling on error detection (for SLOs) but that's more advanced.
Given the complexity, we might start with a simpler version that only uses:
-
Current configuration (sampling rates, retention, and cardinality numbers)
-
Tenant constraints (min sampling, min retention, protected labels)
-
And then make suggestions based on:
- If sampling is 100% and the volume is above a threshold, suggest reducing to 50% (or a step) if allowed.
- If cardinality is very high and there are no protected labels, suggest reducing by removing the top N labels with the most distinct values? (But that might break things)
However, without query logs, we might be breaking things.
-
Conclusion:
The key to a safe optimizer is to have as much context as possible (query logs, alert rules, dashboard definitions) to avoid breaking the tenant's monitoring.
We should design the system to be extensible so that we can add more data sources (like alert rules) as they become available.
Let's design the core components:
Tenant: - id - name - constraints: min_sampling_rate: float (per service? or global?) min_retention_days: int (per data type?) protected_labels: list of label names (for metrics/logs) that cannot be removed critical_services: list of service names that should not have sampling reduced
Service: - name - data_type: metrics, logs, traces
CurrentState (for a tenant and a service and data_type): - sampling_rate: float - retention_days: int - cardinality: dict (label_name -> distinct_count) - volume_per_day: float (bytes or events)
QueryLog (for a tenant, for a data_type, for a time range): - We might store aggregated info: * For each label, was it used in a query? (boolean) over the last 30 days. * For time ranges: what is the max time range queried? (e.g., 99% of queries are within 7 days)
Opportunity: - tenant_id - service_name - data_type - change_type: 'sampling', 'cardinality', 'retention' - current_value - suggested_value - estimated_savings (monthly) - risk: string (low, medium, high) or description - reason: why we think this is safe?
We run the analysis and then generate opportunities.
-
Example code structure:
class TenantOptimizer: def init(self, tenant_id, observability_client, cost_model): self.tenant_id = tenant_id self.observability_client = observability_client self.cost_model = cost_model self.config = self.load_tenant_config()
def load_tenant_config(self): # load from DB pass def collect_current_state(self): # for each service and data_type, get: # sampling_rate, retention_days, cardinality, volume_per_day pass def collect_query_logs(self, last_days=30): # get aggregated query logs for the tenant for the last_days pass def analyze(self): current_state = self.collect_current_state() query_logs = self.collect_query_logs() opportunities = [] for service, data_type, state in current_state: # Sampling analysis if state.sampling_rate > self.config.min_sampling_rate and service not in self.config.critical_services: # We can try reducing sampling by steps until we hit min or until the reduction would cause too much volume drop? # Actually, we want to reduce as much as possible to save cost, but we might have a step of 10%? # Let's try: if current is 100%, try 50%? But we need to check if that's allowed (min_sampling_rate might be 10%). # We can calculate the new volume and cost for a few steps and pick the one with max savings without going below min and without breaking SLOs? # Without SLO info, we just do: new_sampling = max(min_sampling_rate, current_sampling * 0.5) but that might be too aggressive. # Alternatively, we can have a rule: if volume_per_day > X, then reduce to 10% or 1%? But we must be cautious. # Let's assume we have a function that estimates the impact on SLOs? If not, we just risk. # For now, we do: if current_sampling > 0.5, suggest 0.5; if current_sampling > 0.1, suggest 0.1; but only if above min. # But we also need to consider: if the service is already sampled, we might not want to reduce further if it's already low. # Let's set a threshold: only suggest if current_sampling > 0.5 and min_sampling_rate <= 0.1. if state.sampling_rate > 0.5 and self.config.min_sampling_rate <= 0.1: new_sampling = 0.1 # But wait, we might want to step down gradually? The tenant might not want to go from 100% to 10% in one step. # We can suggest multiple steps? But we are generating one suggestion per change. # We can calculate the saving: new_volume = state.volume_per_day * (new_sampling / state.sampling_rate) saving = self.cost_model.calculate_saving(state.volume_per_day, new_volume, state.retention_days) opportunities.append(Opportunity( tenant_id=self.tenant_id, service_name=service, data_type=data_type, change_type='sampling', current_value=state.sampling_rate, suggested_value=new_sampling, estimated_savings=saving, risk='medium', # because we don't know about SLOs reason='High volume and sampling rate above 50%' )) # Cardinality analysis for label, distinct_count in state.cardinality.items(): if label in self.config.protected_labels: continue # Check if this label is used in any query in the last 30 days if not query_logs.is_label_used(label, service, data_type): # We can remove this label # But removing a label will reduce the cardinality by (distinct_count - 1) because we are removing one dimension. # Actually, if we remove a label from a metric, the number of time series for that metric will be divided by the distinct_count of that label? Not exactly. # The total cardinality reduction is: current_total_series / distinct_count? # Actually, the total cardinality is the product of the distinct counts of all labels. So if we remove one label, the new cardinality is current_total_series / distinct_count. # But we don't have the total cardinality for the metric? We have per-label distinct_count, but we need the product of all labels? # Actually, we are storing the cardinality as a dict of label -> distinct_count. But the total cardinality is the product of all these distinct counts? # Not exactly, because the labels are combined. So the total number of series for a metric is the product of the distinct counts of all labels? # Yes, for a metric with labels (a, b, c) and distinct counts (da, db, dc), the total series is da*db*dc. # So if we remove label 'a', then the new total series is (da*db*dc) / da = db*dc. # But note: we are only looking at one label at a time. We might want to consider removing multiple labels? But we are doing one at a time. # We need to know the total cardinality for the metric? Actually, we are given the cardinality as a dict per label, but we don't have the product? # Wait, the `cardinality` we collected might be the number of active series for each label? That doesn't make sense. # Let's clarify: In Prometheus, the cardinality of a metric is the number of unique time series (each combination of label values). # We can get the total number of series for a metric by querying `count({__name__="metric"})` but that's expensive. # Alternatively, we might have a way to get the total cardinality per metric from the observability platform. # So we need to adjust: we should have for each metric: # metric_name: string # total_series: int # labels: dict of label_name -> distinct_count # Then, if we remove a label with distinct_count = d, the new total_series = total_series / d. # But note: if we remove multiple labels, we divide by the product of their distinct counts. # So in our current state, we should have for each service and data_type, a list of metrics, each with: # name, total_series, labels (dict) # Then, for each label in a metric, if it's not used and not protected, we can suggest removing it. # We need to adjust the code above. # Let's assume we have: # current_state[service][data_type] = { # 'metrics': [ # { # 'name': 'http_requests_total', # 'total_series': 1000, # 'labels': {'method': 5, 'path': 50, 'status': 5} # }, # ... # ] # } # Then, for each metric and each label, we can compute the saving. # We'll adjust the code accordingly. pass # We'll rewrite the cardinality part below. # Retention analysis # We have query_logs that tell us the time range of queries. # For the data_type (logs), we might have a max time range that is used. # If current retention is > max_time_range and current retention > min_retention_days, then we can suggest reducing. # But note: retention is usually set per data type, not per service? Actually, it can be per stream? But let's assume per data type for the tenant. # However, in our current_state, we have retention per service and data_type? That might be too granular. Maybe it's global per data type. # Let's assume we have a global retention for logs and another for metrics. So we might have: # current_state[service][data_type] has retention_days, but if it's the same for all services, we can just look at one. # We can do: for data_type in ['logs', 'metrics', 'traces']: # retention = current_state[any_service][data_type].retention_days # assuming same for all services # max_time_range = query_logs.get_max_time_range(data_type) # if retention > max_time_range and retention > self.config.min_retention_days[data_type]: # new_retention = max_time_range # or maybe min_retention_days[data_type] if that's higher? # saving = self.cost_model.calculate_retention_saving(retention, new_retention, volume_per_day_for_data_type) # opportunities.append(...) return opportunities -
Given the complexity and the fact that we don't have a real observability platform to test, we'll outline the design and leave the implementation as a TODO.
-
However, we can write a simplified version that assumes:
-
We have a way to get for each metric: total_series and labels with distinct counts.
-
We have query logs that tell us for each label and data type if it was used in the last 30 days.
-
We have a cost model that can compute savings for:
- Sampling: saving = (current_sampling - new_sampling) * volume_per_day * cost_per_gb * retention_days / 30? Actually, cost is usually per GB per month. So if we reduce the daily ingestion by X GB, then monthly saving = X * 30 * cost_per_gb.
- Cardinality: if we remove a label, the new total_series = old_total_series / distinct_count. Then the new volume per day for that metric = old_volume_per_day * (new_total_series / old_total_series). But note: the volume per day for a metric is proportional to the number of series? Not exactly, because each sample has the same size? In Prometheus, each sample is about the same size, so yes, volume is proportional to the number of series times the number of samples per series per day? Actually, in Prometheus, the number of samples per series per day depends on the scrape interval. So if we remove a label, we reduce the number of series, and each series still has the same number of samples. So the volume reduction is proportional to the reduction in series.
But note: if we remove a label, we are not removing the metric, we are just aggregating. So the total volume for the metric becomes: old_volume * (new_total_series / old_total_series).
Then the saving for that metric is: old_volume - new_volume, and then multiplied by cost per GB and by 30 days.
However, note that the cost might also depend on cardinality in a different way (like active series cost). So we need to know the pricing model.
-
Retention: saving = (current_retention - new_retention) * volume_per_day * cost_per_gb_per_month? Actually, storage cost is per GB per month. So if we reduce retention by R days, then we are deleting R days of data. But note: the data is stored for the entire retention period. So if we reduce from 30 to 7, we are not just saving 23 days of storage, but we are reducing the total stored data by (23/30) of the current stored data? Actually, the stored data is the accumulation of the last 30 days. If we change to 7 days, then after 30 days, we will only have 7 days of data. But the saving is immediate? We delete old data, so we free up storage.
The monthly cost for storage is based on the average amount of data stored in the month. If we change retention in the middle of the month, it's tricky. But for estimation, we can assume that the storage cost is proportional to the retention period (if the daily ingestion is constant). So saving = (current_retention - new_retention) / current_retention * current_storage_cost? Or more directly:
Let daily_ingestion = D (GB/day) Then stored data = D * retention_days (if we assume no compression and steady state). Monthly storage cost = (D * retention_days) * cost_per_gb_per_month? Wait, cost_per_gb_per_month is per GB stored per month. So if we store Dretention_days GB, then the monthly cost is (Dretention_days) * cost_per_gb_per_month? That doesn't sound right because if we store for 30 days, we have 30*D GB stored, and we pay for that every month? Yes, that's how storage cost works.
So if we change retention from R_old to R_new, then the new stored data = D * R_new. The saving per month = (R_old - R_new) * D * cost_per_gb_per_month.
But note: the daily ingestion D might be different for each data type.
-
-
Let's define:
CostModel:
- cost_per_gb_per_month: float
- active_series_cost_per_month: float (optional, if the platform charges per active series)
Then for a given service and data_type, we have:
current: - daily_ingestion_gb: float - retention_days: int - total_series: int (for metrics, for logs it might be different? but we can have a similar concept of streams)
current_monthly_cost = (daily_ingestion_gb * retention_days) * cost_per_gb_per_month + (total_series * active_series_cost_per_month if applicable)
But note: the ingestion cost might be separate from storage? Usually, you pay for ingestion and then separately for storage. But let's assume the cost_per_gb_per_month covers both? Or we have:
cost = ingestion_cost_per_gb + storage_cost_per_gb_per_month
But the problem says "cost optimizer" so we need to know the pricing model.
For simplicity, let's assume:
monthly_cost = (daily_ingestion_gb * 30) * ingestion_cost_per_gb + (daily_ingestion_gb * retention_days) * storage_cost_per_gb_per_month
But note: the ingestion cost is for the data you send in, and storage is for the data you keep.
Alternatively, some platforms (like Datadog) have a combined metric: you pay for the ingested data (per GB) and then storage is included for a certain period? It varies.
We need to be flexible.
-
Given the time, we'll design an abstract CostModel that the user can implement based on their pricing.
We'll define:
class CostModel: def ingestion_cost(self, daily_ingestion_gb): # returns monthly ingestion cost for the given daily ingestion (assuming 30 days) pass
def storage_cost(self, daily_ingestion_gb, retention_days): # returns monthly storage cost for the given daily ingestion and retention pass def active_series_cost(self, total_series): # returns monthly cost for active series (if applicable) passThen the total monthly cost = ingestion_cost(daily_ingestion_gb) + storage_cost(daily_ingestion_gb, retention_days) + active_series_cost(total_series)
-
Now, for sampling change:
new_daily_ingestion = current_daily_ingestion * (new_sampling / current_sampling)
Then the new monthly cost = ingestion_cost(new_daily_ingestion) + storage_cost(new_daily_ingestion, retention_days) + active_series_cost(new_total_series) # but note: active series might also change? In Prometheus, active series are the number of time series being scraped. If we sample, we are reducing the number of samples per series? Actually, no: sampling in the context of Prometheus usually means we sample the targets (scrape less often) or we use remote write sampling?
Wait: there are two types of sampling: a) Scrape interval sampling: we scrape less often, so we get fewer samples per series. This reduces the ingestion volume and also the stored data (because we have fewer samples). But the number of active series remains the same. b) Cardinality reduction: we remove labels, which reduces the number of series.
In our case, we are talking about sampling as in reducing the percentage of data we collect? For example, if we have a service that emits 1000 logs per second, we might only send 100 (10%) to the observability platform. This reduces the volume and also the cardinality? Actually, if we sample logs, we might reduce the number of distinct log streams? Not necessarily, because we are sampling at the source. So the cardinality (number of distinct label sets) might remain the same if we sample uniformly? Actually, if we sample uniformly, we might lose some rare label combinations, so cardinality might drop a little, but not as much as removing a label.
So for sampling, we assume that the number of series (for metrics) or streams (for logs) remains the same, but the number of samples per series is reduced? Actually, if we sample at the source, we are reducing the number of events, which for logs means we have fewer log entries. The number of distinct label combinations (streams) might reduce if we sample, but it's not a straightforward reduction.
Given the complexity, for sampling we will only consider the reduction in daily ingestion volume and assume that the number of series remains the same? But that might not be accurate.
Alternatively, if we are using Prometheus, sampling might be done by reducing the scrape frequency. Then the number of series remains the same, but the number of samples per series per day is reduced.
So let's assume:
For metrics: current_samples_per_series_per_day = 86400 / scrape_interval_seconds new_samples_per_series_per_day = current_samples_per_series_per_day * (new_sampling / current_sampling) Then the new daily ingestion = current_daily_ingestion * (new_sampling / current_sampling) And the total_series remains the same.For logs: if we sample at the source (like Fluentd sampling), then we are reducing the number of log entries. The number of log streams (distinct label sets) might not change because we are still sending logs for the same set of services, but just fewer logs per stream. So the volume reduction is proportional.
So for simplicity, we'll assume:
new_daily_ingestion = current_daily_ingestion * (new_sampling / current_sampling) and total_series remains the same for metrics (because we are not changing the number of time series) and for logs, we might have a reduction in the number of log streams? Actually, no: if we sample logs, we are reducing the number of log events, but each log event still has the same labels? So the number of distinct label combinations (streams) might remain the same because we are still logging from the same services, but just fewer logs. However, if we sample, we might miss some rare labels? But it's not a deterministic reduction.Given the uncertainty, for log sampling we might not change the cardinality. So we treat it the same as metrics.
But wait: in Loki, logs are stored by stream (label set). If we sample, we are still storing the same number of streams? Actually, if we sample at the source, we are sending fewer log entries for each stream. So the number of streams remains the same.
So the saving from sampling is only in the volume (and thus storage) and not in the active series cost.
Therefore, for sampling change:
new_daily_ingestion = current_daily_ingestion * (new_sampling / current_sampling) new_total_series = current_total_series (unchanged)Then the cost saving = (current_monthly_cost - new_monthly_cost)
where: current_monthly_cost = cost_model.ingestion_cost(current_daily_ingestion) + cost_model.storage_cost(current_daily_ingestion, retention_days) + cost_model.active_series_cost(current_total_series)
new_monthly_cost = cost_model.ingestion_cost(new_daily_ingestion) + cost_model.storage_cost(new_daily_ingestion, retention_days) + cost_model.active_series_cost(current_total_series)Note: retention_days remains the same.
For cardinality change (removing a label):
We assume we are removing one label from a metric (or log stream). Then:
old_total_series = total_series for that metric distinct_count = distinct_count of the label we are removing new_total_series = old_total_series / distinct_count (if we remove that label, we aggregate over that label) But note: the daily_ingestion for that metric will also change? The number of samples per series remains the same? For metrics, each series still has the same number of samples. So the new daily_ingestion for that metric = current_daily_ingestion_for_metric * (new_total_series / old_total_series) However, if the metric is part of a larger service, we need to adjust the service's total daily_ingestion and total_series. Let's assume we have a service with multiple metrics. We are going to adjust only the metric we are changing. So for the service: new_daily_ingestion = current_daily_ingestion - (current_daily_ingestion_for_metric * (1 - new_total_series/old_total_series)) new_total_series = current_total_series - old_total_series + new_total_series But note: the active_series_cost is based on the total_series of the service? Or per metric? Usually, the platform charges per active series (time series) across all metrics. So we adjust the service's total_series and daily_ingestion.Then compute the cost saving for the service.
For retention change:
new_retention = suggested_retention daily_ingestion remains the same? Actually, the daily ingestion is the same, but we are storing less data.
So:
new_storage_cost = cost_model.storage_cost(daily_ingestion, new_retention) saving = current_storage_cost - new_storage_costBut note: the ingestion cost remains the same because we are still ingesting the same amount.
So:
current_monthly_cost = ingestion_cost(daily_ingestion) + storage_cost(daily_ingestion, current_retention) + active_series_cost(total_series) new_monthly_cost = ingestion_cost(daily_ingestion) + storage_cost(daily_ingestion, new_retention) + active_series_cost(total_series) saving = current_monthly_cost - new_monthly_cost -
Given the above, we can implement the analyzer for cardinality only if we have the per-metric breakdown.
Let's redefine the current state:
For each tenant, we have a list of services.
For each service, we have: - daily_ingestion_gb: float (total for the service) - total_series: int (total for the service, across all metrics) - metrics: list of MetricInfo
MetricInfo: - name: string - daily_ingestion_gb: float (or we can compute from the total and the proportion? But we need per-metric for cardinality changes) - total_series: int - labels: dict of label_name -> distinct_count
How do we get per-metric daily_ingestion? We might not have that. We might only have the total for the service.
Without per-metric breakdown, we cannot do cardinality reduction per label because we don't know how much each metric contributes.
So we need to collect per-metric data. That might be expensive.
Alternatively, we can estimate: if we know the total_series for the service and the product of distinct counts for each label for each metric, we can compute the total_series for each metric? Actually, the total_series for the service is the sum of the total_series for each metric.
But we don't have the total_series per metric. We only have the labels and their distinct counts for each metric? Actually, we can get the total_series for a metric by querying the observability platform:
count(metric_name). So we can get per-metric total_series.And we can get the daily ingestion per metric? Not directly, but we can estimate by the number of samples? In Prometheus, each sample is about the same size. So the daily ingestion for a metric is proportional to the number of samples per day, which is (total_series * samples_per_series_per_day). And samples_per_series_per_day is 86400 / scrape_interval. But if we have multiple metrics with different scrape intervals? Usually, all metrics are scraped at the same interval.
So we can assume:
daily_ingestion_for_metric = (total_series_for_metric * samples_per_series_per_day) * sample_sizeBut we don't know the sample_size. However, we can compute the proportion:
total_daily_ingestion = sum( daily_ingestion_for_metric for all metrics ) So if we know the total_daily_ingestion and the total_series for the service and the total_series for each metric, we can compute: daily_ingestion_for_metric = total_daily_ingestion * (total_series_for_metric / total_service_series)This assumes that all samples have the same size. That's approximately true for metrics.
So we can do:
total_service_series = sum(metric.total_series for metric in metrics) for metric in metrics: metric.daily_ingestion_gb = total_daily_ingestion * (metric.total_series / total_service_series)Then we can compute the effect of removing a label from a metric.
-
Given the above, we can proceed.
-
However, note that the above analysis is for metrics. For logs and traces, the structure might be different.
For logs (like in Loki), we have streams (label sets). Each stream has a volume of logs. The total ingestion is the sum of the volumes of all streams.
If we remove a label from a log stream, we are aggregating streams that differ only in that label. So:
old_streams: each stream has a label set. We group by all labels except the one we remove. Then the new stream for a group is the sum of the volumes of the old streams in that group.
The new_total_streams = number of groups.
The new_total_ingestion = sum of the volumes of the new streams (which is the same as the old total ingestion? Actually, no: we are not changing the log volume, we are just aggregating. So the total ingestion remains the same?
Wait: if we remove a label, we are changing the label set of the logs. The log message itself might contain that label? In Loki, the labels are extracted from the log line. If we remove a label from the stream selector, we are not removing that label from the log line? Actually, in Loki, each log line is stored with a set of labels. If we remove a label from the stream, we are essentially dropping that label from the set of labels for that log line?
This is confusing.
In practice, when we say "remove a label" for logs, we mean we change the logging configuration to not extract that label. Then the logs will no longer have that label. So the number of distinct label sets (streams) will reduce because we are not distinguishing by that label.
And the volume of logs (number of log lines) remains the same? Yes, because we are still logging the same number of lines, but the label set is smaller.
So the total ingestion (number of log lines) remains the same, but the number of streams reduces.
In Loki, the cost might be based on the number of streams and the volume? Actually, Loki charges by the volume of ingested data (GB) and the number of streams? I think it's only by volume. But some platforms charge by active streams.
So we need to know the pricing.
For simplicity, let's assume that removing a label for logs does not reduce the ingestion volume (because we are still sending the same logs) but reduces the cardinality (number of streams). If the platform charges per stream, then we save on the active series cost.
But if the platform only charges by volume, then removing a label doesn't save money? Actually, it might save a little because the label values are stored in the index, so less index data? But that's negligible.
So for logs, cardinality reduction might only save on active series cost, not on ingestion volume.
This is getting very platform-specific.
-
Given the complexity and the fact that we don't have a specific platform, we might limit our initial version to metrics only, and assume:
- The platform charges by:
- Ingested data volume (GB/month)
- Active series (per month)
And we ignore storage cost because it's included in the ingestion cost? Or we assume storage is included.
Actually, let's assume the cost is:
monthly_cost = (daily_ingestion_gb * 30) * cost_per_gb + (total_series) * cost_per_active_series_per_monthThen:
Sampling change: reduces daily_ingestion_gb, but total_series remains the same. Cardinality change: reduces total_series, but daily_ingestion_gb remains the same? Wait, if we remove a label, we are aggregating series, so the number of samples per day (which is total_series * samples_per_series_per_day) reduces? Actually, no: we are still collecting the same number of samples, but now they are aggregated. So the number of samples per day remains the same? In Prometheus, if we remove a label, we are changing the label set of the time series. The underlying data (the samples) are still there, but they are now stored in fewer time series. So the number of samples is the same, but the number of time series is reduced. However, the storage of samples: each sample is stored in a time series. So if we have fewer time series, we have fewer samples? No, we have the same number of samples but they are grouped. Actually, if we remove a label, we are summing (or aggregating) the samples that had the same everything except that label. So the total number of samples is reduced? Let's think: Example: We have a metric `http_requests_total` with labels `method` and `path`. Suppose we have: method=GET, path=/a: 10 samples method=GET, path=/b: 20 samples method=POST, path=/a: 5 samples method=POST, path=/b: 15 samples If we remove the `path` label, we get: method=GET: 30 samples method=POST: 20 samples So the total number of samples is the same? Actually, we had 10+20+5+15 = 50 samples. Now we have 30+20 = 50 samples. So the number of samples is the same. But in Prometheus, each sample is stored in a time series. Originally, we had 4 time series. After removing `path`, we have 2 time series. But each time series now has the same number of samples? Actually, the samples are aggregated. So the `method=GET` time series has the sum of the two original time series for each timestamp? So we have the same number of timestamps, but the value is the sum. So the storage for the samples: we store the same number of samples (same number of (timestamp, value) pairs) but in fewer time series? Actually, each time series is a set of (timestamp, value) pairs. When we aggregate, we are creating a new time series that has the same timestamps? Not exactly: if we use `sum by (method)`, we get a new time series for each `method` that has the same timestamps as the original? Actually, the timestamps are the same because we are summing at the same timestamps. So the number of sample points is the same. But the number of time series is reduced. Therefore, the volume of data stored (the samples) remains the same? But we have fewer time series, so the index (which maps label sets to time series) is smaller. So the cost saving from cardinality reduction is only in the active series cost, not in the storage of samples.But wait: in some systems, like Thanos or Cortex, they might store the samples in a way that the number of time series affects the number of blocks? I'm not sure.
Given the uncertainty, we will assume:
- Ingested data volume is proportional to the number of samples (which is total_series * samples_per_series_per_day). And if we remove a label, the number of samples remains the same? Actually, we are aggregating, so the number of samples (data points) is the same? Let me re-examine: when we remove a label, we are effectively summing over that label. The resulting time series have the same timestamps as the original? In Prometheus, if you do `sum by (method) (http_requests_total)`, you get a time series for each method that has the same timestamps as the original? Actually, the original time series might have different timestamps? No, they are scraped at the same time, so they have the same timestamps. So the aggregated time series will have the same timestamps. So the total number of samples (timestamp, value pairs) is the same. Therefore, the storage for the samples is the same. Only the index (the label sets) is smaller.So if the platform charges by the volume of samples (which is the same) and by the number of active series (which is reduced), then we save on active series cost.
But if the platform charges only by the volume of data (including index? usually index is small), then we don't save much.
So for cardinality reduction, we only consider the active series cost saving.
Then:
new_total_series = old_total_series / distinct_count (for that label, assuming we remove it from one metric and that label's distinct_count is d) But note: the total_series for the service is the sum of the total_series for each metric. So if we remove a label from one metric, we reduce the total_series for that metric by a factor of d, and then the service total_series reduces by (old_series_metric - new_series_metric).Then the saving in active series cost = (old_service_total_series - new_service_total_series) * cost_per_active_series_per_month.
And the ingestion and storage cost remain the same? Because the number of samples is the same?
But wait: if we remove a label, we are changing the way the data is stored. In Prometheus, when you remove a label, you are effectively rewriting the time series. The number of samples might change if the original time series had different timestamps? We assumed they are the same.
Given the complexity, we'll assume that removing a label does not change the number of samples, so the ingestion and storage volume remains the same.
Therefore, the only saving is from active series cost.
For retention change:
We reduce the storage of samples. The number of samples stored is reduced because we are keeping fewer days.
The number of active series? The active series are the series that are currently being scraped (or sent). That doesn't change with retention. So active series cost remains the same.
So the saving is only in storage cost.
But note: if the platform charges for ingestion separately, that doesn't change because we are still ingesting the same amount per day.
So:
new_storage_cost = cost_model.storage_cost(daily_ingestion_gb, new_retention) saving = current_storage_cost - new_storage_costAnd current_storage_cost = cost_model.storage_cost(daily_ingestion_gb, current_retention)
And the ingestion cost remains the same.
- The platform charges by:
-
Summary of assumptions:
-
Cost model: monthly_cost = ingestion_cost(daily_ingestion) + storage_cost(daily_ingestion, retention) + active_series_cost(total_series)
-
Sampling change: new_daily_ingestion = current_daily_ingestion * (new_sampling / current_sampling) new_total_series = current_total_series (unchanged) saving = (current_monthly_cost - new_monthly_cost)
-
Cardinality change (removing one label from one metric): We assume we have per-metric data. For the metric we change: old_metric_series = M_old d = distinct_count of the label new_metric_series = M_old / d
Then: new_service_total_series = current_service_total_series - M_old + new_metric_series new_service_daily_ingestion = current_service_daily_ingestion (unchanged, because number of samples is same)
Then the new monthly cost for the service: ingestion_cost = cost_model.ingestion_cost(new_service_daily_ingestion) # same as before storage_cost = cost_model.storage_cost(new_service_daily_ingestion, retention_days) # same active_series_cost = cost_model.active_series_cost(new_service_total_series)
saving = (current_service_monthly_cost - new_service_monthly_cost)
-
Retention change: new_retention = suggested_retention new_daily_ingestion = current_daily_ingestion (unchanged) new_total_series = current_total_series (unchanged) saving = current_storage_cost - new_storage_cost
-
-
Implementation:
We'll design the classes accordingly.
However, note: in the cardinality change, we are only removing one label at a time. But we might want to remove multiple labels? We can run multiple simulations, but then the effects are not independent because removing two labels from the same metric would reduce the series by a factor of (d1 * d2). But we can do one at a time and then the tenant can apply multiple.
Also, we must check that after removing one label, the remaining labels are not protected? Actually, we check each label independently. But if we remove one label, the other labels remain. So we can suggest removing multiple labels in Separate suggestions.
-
Given the time, we'll write a high-level code structure and leave the details.
We assume we have:
class Tenant: id: str config: TenantConfig
class TenantConfig: min_sampling_rate: float = 0.1 min_retention_days: int = 7 protected_labels: List[str] = [] critical_services: List[str] = []
class ServiceState: service_name: str daily_ingestion_gb: float total_series: int retention_days: int metrics: List[MetricInfo]
class MetricInfo: name: str total_series: int labels: Dict[str, int] # label_name -> distinct_count
class QueryLogs: # For each label, for each data_type, was it used in the last 30 days? # We might have a set: used_labels[data_type][label_name] = True/False # And for retention: max_time_range[data_type] = int (days)
class CostModel: def ingestion_cost(self, daily_ingestion_gb: float) -> float: # returns monthly cost pass
def storage_cost(self, daily_ingestion_gb: float, retention_days: int) -> float: pass def active_series_cost(self, total_series: int) -> float: passclass Opportunity: tenant_id: str service_name: str data_type: str # 'metrics', 'logs', etc. change_type: str # 'sampling', 'cardinality', 'retention' current_value: Any suggested_value: Any estimated_savings: float risk: str reason: str
Then the analyzer:
class Analyzer: def init(self, tenant: Tenant, service_states: List[ServiceState], query_logs: QueryLogs, cost_model: CostModel): self.tenant = tenant self.service_states = service_states self.query_logs = query_logs self.cost_model = cost_model
def analyze(self) -> List[Opportunity]: opportunities = [] for service_state in self.service_states: # Skip if service is critical for sampling changes? # We'll handle sampling for the whole service? Or per metric? Usually sampling is set per service or per job. # Sampling analysis: we assume the service has a sampling rate? But we don't have per-service sampling in the state? # We need to adjust ServiceState to include sampling_rate? Yes. # Let's add to ServiceState: # sampling_rate: float # Then: if service_state.sampling_rate > self.tenant.config.min_sampling_rate and service_state.service_name not in self.tenant.config.critical_services: # We'll suggest reducing to 10% if current is > 50%, or to 50% if current is between 50% and 10%? # We can have a step function. if service_state.sampling_rate > 0.5: new_sampling = 0.1 elif service_state.sampling_rate > 0.1: new_sampling = 0.1 else: new_sampling = service_state.sampling_rate # no change if new_sampling < service_state.sampling_rate: # Calculate new daily ingestion new_daily_ingestion = service_state.daily_ingestion_gb * (new_sampling / service_state.sampling_rate) new_total_series = service_state.total_series # unchanged current_cost = (self.cost_model.ingestion_cost(service_state.daily_ingestion_gb) + self.cost_model.storage_cost(service_state.daily_ingestion_gb, service_state.retention_days) + self.cost_model.active_series_cost(service_state.total_series)) new_cost = (self.cost_model.ingestion_cost(new_daily_ingestion) + self.cost_model.storage_cost(new_daily_ingestion, service_state.retention_days) + self.cost_model.active_series_cost(new_total_series)) saving = current_cost - new_cost if saving > 0: opportunities.append(Opportunity( tenant_id=self.tenant.id, service_name=service_state.service_name, data_type='metrics', # we assume sampling change is for metrics? Actually, it could be for logs too. change_type='sampling', current_value=service_state.sampling_rate, suggested_value=new_sampling, estimated_savings=saving, risk='medium', reason=f'High sampling rate ({service_state.sampling_rate*100}%) and service is not critical.' )) # Cardinality analysis: per metric for metric in service_state.metrics: for label, distinct_count in metric.labels.items(): if label in self.tenant.config.protected_labels: continue if not self.query_logs.is_label_used(label, 'metrics'): # we assume the same for all services? We might need per-service and data_type. # We can remove this label. # Calculate new total_series for this metric old_metric_series = metric.total_series new_metric_series = old_metric_series // distinct_count # integer division, but it might not be exact? But we assume the distinct_count is exact. # Then new service total_series: new_service_total_series = service_state.total_series - old_metric_series + new_metric_series current_cost = (self.cost_model.ingestion_cost(service_state.daily_ingestion_gb) + self.cost_model.storage_cost(service_state.daily_ingestion_gb, service_state.retention_days) + self.cost_model.active_series_cost(service_state.total_series)) new_cost = (self.cost_model.ingestion_cost(service_state.daily_ingestion_gb) + self.cost_model.storage_cost(service_state.daily_ingestion_gb, service_state.retention_days) + self.cost_model.active_series_cost(new_service_total_series)) saving = current_cost - new_cost if saving > 0: opportunities.append(Opportunity( tenant_id=self.tenant.id, service_name=service_state.service_name, data_type='metrics', change_type='cardinality', current_value=f'Label {label} with {distinct_count} distinct values', suggested_value='Remove label', estimated_savings=saving, risk='high', # because we might break queries reason=f'Label "{label}" is not used in any query in the last 30 days and has {distinct_count} distinct values.' )) # Retention analysis: we assume retention is set per data type and is the same for all services? # Actually, in our ServiceState, we have retention_days per service? That might be too granular. # Usually, retention is set per data type (metrics, logs) globally. But let's assume it's per service. # We'll do retention only once for the tenant? Because changing retention for one service might not be possible if it's global. # Let's assume retention is global for the tenant for a data type. So we can do: # We'll skip retention in the service loop and do it after. pass # Retention analysis (for the whole tenant, for each data type) # We need to know the daily ingestion for each data type. We can sum over services for that data type. # But our service_state might be for a specific data type? Actually, we have service_state for each service and each data type? # We need to adjust: we might have multiple ServiceState for the same service but different data_type? Or we have one ServiceState per service that aggregates all data types? # Given the complexity, let's assume we are only dealing with metrics in this example. # We'll skip retention for now. return opportunities -
This is a very simplified version.
-
We must also consider that the tenant might have multiple data types (metrics, logs, traces) and each has different pricing and constraints.
We would need to run the analysis per data type.
-
Given the time, we'll stop here and note that the above is a proof of concept.
-
We also need to consider that the suggestions might be interdependent. For example, if we reduce sampling, the cardinality might change (because we might see fewer distinct label combinations). But we are not modeling that.
-
Conclusion:
This is a complex problem that requires deep integration with the observability platform and a good understanding of the tenant's usage patterns.
We have outlined a design that uses:
- Tenant constraints to avoid breaking requirements.
- Query logs to find unused labels and retention needs.
- Cost modeling to estimate savings.
But without real data and a specific platform, it's hard to implement.
We can build a prototype that works with Prometheus and Grafana Loki, for example.
-
Next steps:
- Define the data model for the tenant's current state (including per-metric breakdown for metrics, and per-stream for logs?).
- Implement the integration with Prometheus to get the per-metric series count and label distinct counts.
- Implement the integration with Loki to get per-stream cardinality and label distinct counts? That might be expensive.
- Implement query log analysis: for Prometheus, we can use the query log from Cortex or Thanos? Or from Grafana? We might not have access.
- Implement the cost model based on the vendor's pricing.
Given the time, we'll leave the implementation as a future task.
-
We'll produce a design document and a simple code skeleton.
However, note that the problem asks to "engineer" a solution. We have provided a design and a code structure.
Let's write a simple code skeleton that demonstrates the classes and the flow, but without the actual implementation of the integration.
We'll assume we have:
- A class
ObservabilityPlatformthat can give us the current state for a tenant. - A class
QueryLogAnalyzerthat can tell us if a label is used.
But we won't implement these.
We'll write:
class MultiTenantCostOptimizer: def init(self, platform_client, cost_model): self.platform_client = platform_client self.cost_model = cost_model
def optimize_tenant(self, tenant_id): tenant_config = self.load_tenant_config(tenant_id) current_state = self.platform_client.get_current_state(tenant_id) query_logs = self.platform_client.get_query_logs(tenant_id, days=30) analyzer = Analyzer(tenant_config, current_state, query_logs, self.cost_model) opportunities = analyzer.analyze() return opportunitiesAnd then the Analyzer class as above.
-
We must note that this is a simplified version and might not be production-ready.
-
Finally, we must consider that the optimizer should run periodically and provide a report to the tenant.
-
We'll also need a way for the tenant to provide feedback on the suggestions (e.g., accept, reject, modify) so that we can learn.
-
Given the scope, we'll stop here. I'll design a comprehensive multi-tenant observability cost optimizer system. This is a complex system that requires careful consideration of tenant isolation, safety constraints, and actionable recommendations.
System Architecture
graph TB subgraph "Data Collection Layer" A[Observability Platform APIs] --> B[Data Ingestion Service] C[Cost/Billing APIs] --> B D[Query Pattern Analyzer] --> B end subgraph "Processing Engine" B --> E[Tenant Context Builder] E --> F[Cost Model Service] E --> G[Usage Analyzer] F --> H[Optimization Engine] G --> H end subgraph "Recommendation Layer" H --> I[Suggestion Generator] I --> J[Safety Validator] J --> K[Tenant-Specific Formatter] end subgraph " Delivery Layer" K --> L[API Gateway] K --> M[Scheduled Reports] K --> N[Alert Integration] end subgraph "Feedback Loop" O[Tenant Implementation Tracker] --> P[Effectiveness Analyzer] P --> Q[ML Model Trainer] Q --> H endCore Components
1. Tenant Context & Constraints Store
class TenantContext: tenant_id: str business_hours: List[ datetime ] # For time-based recommendations critical_services: List[str] # Services with 100% sampling required sla_requirements: Dict[str, float] # SLOs per service budget_limit: Optional[float] # Monthly budget cap compliance_retention: Dict[str, int] # Minimum retention per data type allowed_cardinality_reduction: Dict[str, Set[str]] # Allowed labels to drop per metric max_sampling_reduction: float # e.g., 0.1 = max 90% reduction maintenance_windows: List[TimeRange] notification_channels: List[str]2. Multi-Dimensional Cost Model
class CostModel: def calculate_current_cost(tenant_id: str) -> DetailedCost: """ Returns breakdown: - Ingestion cost (GB-based) - Cardinality cost (active series/dimensionality) - Retention cost (storage) - Query cost (if applicable) """ def simulate_change(tenant_id: str, changes: OptimizationChange) -> CostProjection: """ Projects cost impact of specific changes """3. Usage Analyzer & Pattern Detection
class UsageAnalyzer: def detect_sampling_opportunities(tenant_id: str) -> List[SamplingOpportunity]: """ Analyzes: - Traffic patterns (diurnal, weekly) - Error rates vs sampling - SLO compliance with reduced sampling - Query accuracy needs """ def detect_cardinality_bloat(tenant_id: str) -> List[CardinalityIssue]: """ Identifies: - High-cardinality labels with low query usage - Label combinations never queried - Zombie metrics (no queries in 30 days) - Low-value high-cardinality dimensions """ def analyze_retention_efficiency(tenant_id: str) -> RetentionProfile: """ Determines: - Query time ranges (p99, p95, p50) - Data age distribution of queries - Compliance vs actual needs """4. Optimization Engine with Constraint Solver
class OptimizationEngine: def generate_recommendations(tenant_id: str) -> List[Recommendation]: recommendations = [] # 1. Cardinality reductions (highest ROI) cardinality_ops = self._find_cardinality_opportunities(tenant_id) for op in cardinality_ops: if self._safety_check(tenant_id, op): recommendations.append(self._create_recommendation(op)) # 2. Sampling optimizations sampling_ops = self._find_sampling_opportunities(tenant_id) for op in sampling_ops: if self._safety_check(tenant_id, op): recommendations.append(self._create_recommendation(op)) # 3. Retention adjustments retention_ops = self._find_retention_opportunities(tenant_id) for op in retention_ops: if self._safety_check(tenant_id, op): recommendations.append(self._create_recommendation(op)) # Sort by savings/risk ratio return sorted(recommendations, key=lambda r: r.estimated_monthly_savings / r.risk_score, reverse=True)5. Safety Validator & Impact Simulation
class SafetyValidator: def validate_sampling_change(tenant_id: str, service: str, new_rate: float) -> ValidationResult: """ Checks: - SLO coverage at new sampling rate (statistical simulation) - Alert threshold coverage - Historical incident detection capability - Business criticality of service """ def validate_cardinality_change(tenant_id: str, metric: str, labels_to_remove: Set[str]) -> ValidationResult: """ Checks: - Dashboard/alert usage of these labels (query log analysis) - Debugging patterns (do engineers use these for troubleshooting?) - Downstream system dependencies - Compliance requirements """ def validate_retention_change(tenant_id: str, data_type: str, new_retention: int) -> ValidationResult: """ Checks: - Compliance requirements - Historical query patterns beyond new retention - Audit requirements - Capacity planning needs """6. Intelligent Recommendation Templates
class RecommendationFormatter: def format_recommendation(recommendation: OptimizationRecommendation) -> Dict: return { "id": recommendation.id, "type": recommendation.type, "title": self._generate_title(recommendation), "description": self._generate_description(recommendation), "current_state": recommendation.current_state, "proposed_change": recommendation.proposed_change, "estimated_savings": { "monthly": recommendation.estimated_monthly_savings, "annual": recommendation.estimated_annual_savings, "percentage": recommendation.savings_percentage }, "risk_level": recommendation.risk_level, "risk_factors": recommendation.risk_factors, "implementation_steps": self._generate_steps(recommendation), "rollback_procedure": self._generate_rollback(recommendation), "validation_tests": self._generate_validation_tests(recommendation), "similar_tenants": self._find_similar_adoptions(recommendation), "confidence_score": recommendation.confidence_score }Key Algorithms
1. High-Cardinality Detection
def detect_cardinality_bloat(time_series_data: pd.DataFrame) -> List[CardinalityIssue]: issues = [] # Identify labels with >100 distinct values high_card_labels = {} for label in all_labels: distinct_count = time_series_data[label].nunique() if distinct_count > 100: high_card_labels[label] = distinct_count # Check query usage query_logs = get_query_logs(last_30_days) for label, count in high_card_labels.items(): usage = query_logs[label].query_count if usage < (count * 0.01): # Less than 1% of values queried issues.append(CardinalityIssue( label=label, distinct_values=count, query_coverage=usage/count, estimated_series_reduction=calculate_reduction(label) )) return sorted(issues, key=lambda x: x.estimated_savings, reverse=True)2. SLO-Aware Sampling Simulation
def simulate_sampling_impact(service: str, current_rate: float, proposed_rate: float, slo_threshold: float) -> Tuple[float, float]: """ Returns (slo_compliance_probability, error_detection_capability) Uses binomial distribution and historical error rates """ historical_errors = get_historical_errors(service) error_rate = historical_errors.mean() # Probability of detecting at least one error in a window # Using Poisson approximation to binomial window_size = get_incident_window(service) # e.g., 5 minutes λ_current = error_rate * window_size * current_rate λ_proposed = error_rate * window_size * proposed_rate detection_prob_current = 1 - math.exp(-λ_current) detection_prob_proposed = 1 - math.exp(-λ_proposed) return detection_prob_proposed, detection_prob_current3. Retention Optimization Model
def optimize_retention(query_patterns: pd.DataFrame, min_retention: int, compliance_retention: int) -> RetentionOptimization: """ Analyze query time ranges to find optimal retention """ # Get percentiles of query lookback p50 = query_patterns['lookback_days'].quantile(0.5) p95 = query_patterns['lookback_days'].quantile(0.95) p99 = query_patterns['lookback_days'].quantile(0.99) # Balance cost vs accessibility storage_cost_per_day = get_storage_cost_per_day() # Find sweet spot: covers p95 but not full retention candidate_retentions = [ max(min_retention, math.ceil(p95)), max(min_retention, math.ceil(p99)), compliance_retention # Must meet compliance ] best = min(candidate_retentions, key=lambda r: (r * storage_cost_per_day) / (query_patterns[query_patterns['lookback_days'] <= r].shape[0])) return RetentionOptimization( recommended_retention=best, current_retention=get_current_retention(), savings=(get_current_retention() - best) * 30 * storage_cost_per_day, query_coverage=query_patterns[query_patterns['lookback_days'] <= best].shape[0] / query_patterns.shape[0] )Implementation Strategy
Phase 1: Foundation (Weeks 1-4)
-
Data Pipeline Setup
- Connect to observability platforms (Prometheus, Loki, Tempo, etc.)
- Implement cost data collection from cloud providers
- Build query log aggregation
-
Basic Tenant Isolation
- Multi-tenant data segregation
- Per-tenant configuration storage
- RBAC implementation
Phase 2: Core Analytics (Weeks 5-8)
-
Cardinality Analysis
- High-cardinality label detection
- Query log correlation
- Simple reduction recommendations
-
Sampling Analysis
- Traffic pattern analysis
- Basic SLO impact estimation
- Static recommendation rules
Phase 3: Advanced Features (Weeks 9-12)
-
Safety Validation
- SLO simulation engine
- Query pattern validation
- Rollback procedure generation
-
Retention Optimization
- Query time range analysis
- Compliance-aware recommendations
Phase 4: Intelligence & Automation (Weeks 13-16)
-
ML Enhancement
- Prediction of future cardinality growth
- Anomaly detection in cost patterns
- Tenant similarity clustering
-
Feedback Loop
- Track recommendation adoption
- Measure actual savings vs projected
- Continuous model improvement
-
Integration
- GitOps workflow integration
- API-driven implementation
- ChatOps notifications
Safety Mechanisms
-
Canary Deployment
- Implement changes on 5% of traffic first
- Monitor for 24-48 hours before full rollout
- Automatic rollback on anomaly detection
-
Change Validation Suite
def validate_recommendation_implementation(tenant_id, change): # Pre-implementation checks assert change.respects_sla_requirements() assert change.meets_compliance_requirements() assert change.has_rollback_plan() # Post-implementation monitoring monitor_for = ['error_rate_spike', 'latency_increase', 'alert_misses', 'dashboards_broken'] return monitor_changes(change, monitor_for, window_hours=24)- Approval Workflow
- Critical services → requires manual approval
- Non-critical → auto-approve with monitoring
- High-risk → requires 2-level approval
Monitoring & Observability
The optimizer should monitor itself:
self_monitoring: - recommendation_accuracy: "Compare projected vs actual savings" - safety_violations: "Count of post-change incidents" - adoption_rate: "% of recommendations implemented" - tenant_satisfaction: "CSAT scores from users" - false_positive_rate: "% of recommendations rejected as unsafe"Example Recommendation Output
{ "recommendation_id": "card-2024-01-15-001", "tenant": "acme-corp", "type": "cardinality_reduction", "title": "Remove 'pod_hash' label from kube_pod_info metric", "description": "The 'pod_hash' label has 10,247 distinct values but appears in 0% of queries over the last 30 days. This label provides no diagnostic value as pod names are already uniquely identified by 'pod'.", "current_state": { "metric": "kube_pod_info", "current_series": 524,288, "current_cardinality_contribution": 10,247 }, "proposed_change": { "action": "remove_label", "label": "pod_hash", "from_metrics": ["kube_pod_info", "kube_pod_container_info"] }, "impact": { "estimated_monthly_savings": "$347", "series_reduction": 524,288, "cardinality_reduction_pct": "98%", "risk_level": "low", "slo_impact": "none", "query_impact": "none (0% usage)" }, "implementation": { "steps": [ "1. Update Prometheus relabel config: drop pod_hash", "2. Apply config to all scrape configs", "3. Wait 24h for old series to age out", "4. Verify series count decreased in Grafana" ], "rollback": "Restore original relabel config", "validation_tests": [ "Check all dashboards using kube_pod_info still work", "Verify no alerts triggered on pod metrics", "Confirm series count reduced by ~10,000" ] }, "adoption_stats": { "similar_tenants_adopted": 87, "avg_savings_similar": "$312/month", "issues_reported": 0 } }Technology Stack Recommendations
- Data Processing: Apache Spark/Pandas for large-scale analysis
- Storage: PostgreSQL (tenant configs), ClickHouse (usage analytics)
- ML: Scikit-learn, Prophet (time series forecasting)
- API: FastAPI with async support
- Deployment: Kubernetes with per-tenant namespaces for heavy processing
- Monitoring: The optimizer should use the observability platform it's optimizing!
Critical Considerations
-
Tenant Differentiation
- Startups vs enterprises have different tolerance for risk
- Financial services vs retail have different compliance needs
- Consider tenant maturity (monitoring sophistication)
-
Safety-First Design
- Never recommend changes that could impact SLA/SLO
- Always provide rollback procedures
- Require validation testing before full deployment
-
Gradual Rollout
- Start with non-production tenants
- Begin with low-risk cardinality reductions
- Expand to sampling only after validation
-
Feedback Integration
- Track which recommendations are implemented
- Measure actual vs projected savings
- Use feedback to improve models
This system provides a balanced approach between cost optimization and operational safety, with multiple layers of validation and tenant-specific customization. The key is starting conservatively and building trust through accurate, low-risk recommendations before progressing to more impactful changes.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.