Given: Grafana on premise with prometheus Some anomalies can be detected by viewing a…
- User
Given: Grafana on premise with prometheus
Some anomalies can be detected by viewing a set of charts (slow requests, retries, pending transactions, etc.). SRE operators need to have the opportunity to see all information in one Grafana widget instead of multiple charts to manage incidents. They also need to be familiar with all services.
Abnormal behavior can easily be recognized in one chart. But when trying to find a problem through 10 or more services in one chart, hopping in one graph loses sight of the others, and the problem can't be seen.
The easiest workaround seems to be to convert the linear time series graphs into a single status history widget. I tried the rate() function with a zero, 15-minute, and 24-hour offset. I can also filter events by status: success="yes". For many services, the recognition ability is unacceptable.
Also some problems can't be resolved:
- The detector can occasionally be triggered by problems that happened 24 hours ago
- It doesn't detect zero events gaps as an incidents (should)
- The detector is triggered by events that are not present in the queries
How to solve problem with PromQL with time series?
Assistant
To solve the problem of detecting anomalies and incidents across multiple services in a single, consolidated view within Grafana (backed by Prometheus), while addressing your specific issues:
✅ Goal Recap:
You want:
- A single widget/status history dashboard panel that shows service health over time.
- It should detect anomalies like slow requests, retries, pending transactions, etc.
- It must be able to show all relevant services together without losing context.
- Must avoid false positives from past events and correctly detect zero-event gaps.
🔍 Problem Breakdown
You're currently using
rate()with offsets and filtering by status (success="yes"), but facing these issues:- False positives from 24h-old events – detector triggered by old data.
- Zero-event gaps not detected as incidents – silent failures are missed.
- Events not present in queries still trigger alerts – inconsistency between query and alert logic.
🧠 Solution Strategy
We need to build a PromQL-based anomaly detection system that:
- Detects current issues only (not historical).
- Recognizes absence of metrics as a potential failure.
- Aggregates multi-service statuses into a single time series.
- Outputs a binary or categorical metric per service (e.g.,
0 = healthy,1 = unhealthy).
Then visualize this in Grafana using the Status History Panel or similar.
✅ Step-by-Step Solution
1. Define "Unhealthy" Conditions for Each Metric
Let’s assume you have a metric like
http_requests_total{job="my-service", status=~"5..|4.."}indicating errors.You can define an unhealthy state if there's any error rate above zero in the last N minutes:
( rate(http_requests_total{status=~"5..|4.."}[5m]) > 0 )Or if latency is too high:
( avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1 )Or even retries:
( rate(retries_total[5m]) > 0 )Each of these conditions can be turned into a boolean result (
1if true,0if false).
2. Detect Zero Events (Silent Failures)
If a service stops reporting entirely, use
absent()or check for zero samples:( absent(http_requests_total{job="my-service"}) or (count_over_time(http_requests_total{job="my-service"}[5m]) == 0) )This flags missing metrics as unhealthy.
3. Combine Multiple Anomaly Sources
Use logical ORs to combine all possible unhealthy indicators:
( rate(http_requests_total{status=~"5..|4.."}[5m]) > 0 or avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1 or rate(retries_total[5m]) > 0 or absent(http_requests_total{job="my-service"}) or count_over_time(http_requests_total{job="my-service"}[5m]) == 0 )This gives you a unified indicator for each service.
4. Label It as a Status Series
Wrap it in a comparison so it returns
1for unhealthy,0for healthy:( ( rate(http_requests_total{status=~"5..|4.."}[5m]) > 0 or avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1 or rate(retries_total[5m]) > 0 or absent(http_requests_total{job="my-service"}) or count_over_time(http_requests_total{job="my-service"}[5m]) == 0 ) ) * 1Now you have a numeric time series per service where:
0= healthy1= unhealthy
5. Apply Across All Services
Use
by (job)or another label to apply across services:( ( rate(http_requests_total{status=~"5..|4.."}[5m]) > 0 or avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1 or rate(retries_total[5m]) > 0 or absent(http_requests_total) or count_over_time(http_requests_total[5m]) == 0 ) ) * 1This works if
http_requests_totalhas ajoblabel per service.
6. Visualize Using Grafana Status History Panel
Use the Status History Panel plugin (like alexanderzobnin-zabbix or newer alternatives) or Time Series Panel with
Show as: HeatmaporStacked.Set up the query from step 5 and configure the panel to show:
- X-axis: Time
- Y-axis: Service names
- Color: Based on value (
0= green,1= red)
This will give you a compact, color-coded timeline showing when each service was unhealthy.
7. Avoid False Positives from Past Data
Make sure you're querying only recent data (e.g.,
5mor10m) in the range vector selectors.Also, avoid using
offsetunless necessary — especially if you're trying to compare current behavior only.
8. (Optional) Alerting Rule Example
For real-time alerting, define a recording rule or alerting rule:
groups: - name: service-health-alerts rules: - record: service_unhealthy expr: | ( rate(http_requests_total{status=~"5..|4.."}[5m]) > 0 or avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1 or rate(retries_total[5m]) > 0 or absent(http_requests_total) or count_over_time(http_requests_total[5m]) == 0 ) * 1 - alert: ServiceUnhealthy expr: service_unhealthy == 1 for: 2m labels: severity: warning annotations: summary: "Service {{ $labels.job }} is unhealthy" description: "Service {{ $labels.job }} has shown signs of failure (errors, latency, no data)"
✅ Summary
Feature Solution Combine charts into one view Use PromQL to create a unified health metric per service Detect silent failures Use absent()andcount_over_time()Avoid old event false positives Limit range vectors to short windows (e.g., [5m])Show in Grafana Use Status History or Time Series panel with heatmaps Real-time alerting Define alerting rules based on the same metric
🛠 Tools & Plugins You May Need
- Grafana Plugin: Status History Panel or Time Series Panel with heatmap
- Prometheus Recording Rules: For reusability and performance
- Alertmanager: To manage and route alerts
Would you like me to help write a full example dashboard JSON or alerting config tailored to your metrics?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.