Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b-no-thinking· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

Given: Grafana on premise with prometheus Some anomalies can be detected by viewing a…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-pro’s answer.
  1. User

    Given: Grafana on premise with prometheus

    Some anomalies can be detected by viewing a set of charts (slow requests, retries, pending transactions, etc.). SRE operators need to have the opportunity to see all information in one Grafana widget instead of multiple charts to manage incidents. They also need to be familiar with all services.

    Abnormal behavior can easily be recognized in one chart. But when trying to find a problem through 10 or more services in one chart, hopping in one graph loses sight of the others, and the problem can't be seen.

    The easiest workaround seems to be to convert the linear time series graphs into a single status history widget. I tried the rate() function with a zero, 15-minute, and 24-hour offset. I can also filter events by status: success="yes". For many services, the recognition ability is unacceptable.

    Also some problems can't be resolved:

    • The detector can occasionally be triggered by problems that happened 24 hours ago
    • It doesn't detect zero events gaps as an incidents (should)
    • The detector is triggered by events that are not present in the queries

    How to solve problem with PromQL with time series?

  2. Qwen Assistant

    To solve the problem of detecting anomalies and incidents across multiple services in a single, consolidated view within Grafana (backed by Prometheus), while addressing your specific issues:


    ✅ Goal Recap:

    You want:

    • A single widget/status history dashboard panel that shows service health over time.
    • It should detect anomalies like slow requests, retries, pending transactions, etc.
    • It must be able to show all relevant services together without losing context.
    • Must avoid false positives from past events and correctly detect zero-event gaps.

    🔍 Problem Breakdown

    You're currently using rate() with offsets and filtering by status (success="yes"), but facing these issues:

    1. False positives from 24h-old events – detector triggered by old data.
    2. Zero-event gaps not detected as incidents – silent failures are missed.
    3. Events not present in queries still trigger alerts – inconsistency between query and alert logic.

    🧠 Solution Strategy

    We need to build a PromQL-based anomaly detection system that:

    • Detects current issues only (not historical).
    • Recognizes absence of metrics as a potential failure.
    • Aggregates multi-service statuses into a single time series.
    • Outputs a binary or categorical metric per service (e.g., 0 = healthy, 1 = unhealthy).

    Then visualize this in Grafana using the Status History Panel or similar.


    ✅ Step-by-Step Solution

    1. Define "Unhealthy" Conditions for Each Metric

    Let’s assume you have a metric like http_requests_total{job="my-service", status=~"5..|4.."} indicating errors.

    You can define an unhealthy state if there's any error rate above zero in the last N minutes:

    (
      rate(http_requests_total{status=~"5..|4.."}[5m]) > 0
    )
    

    Or if latency is too high:

    (
      avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1
    )
    

    Or even retries:

    (
      rate(retries_total[5m]) > 0
    )
    

    Each of these conditions can be turned into a boolean result (1 if true, 0 if false).


    2. Detect Zero Events (Silent Failures)

    If a service stops reporting entirely, use absent() or check for zero samples:

    (
      absent(http_requests_total{job="my-service"}) or
      (count_over_time(http_requests_total{job="my-service"}[5m]) == 0)
    )
    

    This flags missing metrics as unhealthy.


    3. Combine Multiple Anomaly Sources

    Use logical ORs to combine all possible unhealthy indicators:

    (
      rate(http_requests_total{status=~"5..|4.."}[5m]) > 0
      or
      avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1
      or
      rate(retries_total[5m]) > 0
      or
      absent(http_requests_total{job="my-service"})
      or
      count_over_time(http_requests_total{job="my-service"}[5m]) == 0
    )
    

    This gives you a unified indicator for each service.


    4. Label It as a Status Series

    Wrap it in a comparison so it returns 1 for unhealthy, 0 for healthy:

    (
      (
        rate(http_requests_total{status=~"5..|4.."}[5m]) > 0
        or
        avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1
        or
        rate(retries_total[5m]) > 0
        or
        absent(http_requests_total{job="my-service"})
        or
        count_over_time(http_requests_total{job="my-service"}[5m]) == 0
      )
    ) * 1
    

    Now you have a numeric time series per service where:

    • 0 = healthy
    • 1 = unhealthy

    5. Apply Across All Services

    Use by (job) or another label to apply across services:

    (
      (
        rate(http_requests_total{status=~"5..|4.."}[5m]) > 0
        or
        avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1
        or
        rate(retries_total[5m]) > 0
        or
        absent(http_requests_total)
        or
        count_over_time(http_requests_total[5m]) == 0
      )
    ) * 1
    

    This works if http_requests_total has a job label per service.


    6. Visualize Using Grafana Status History Panel

    Use the Status History Panel plugin (like alexanderzobnin-zabbix or newer alternatives) or Time Series Panel with Show as: Heatmap or Stacked.

    Set up the query from step 5 and configure the panel to show:

    • X-axis: Time
    • Y-axis: Service names
    • Color: Based on value (0 = green, 1 = red)

    This will give you a compact, color-coded timeline showing when each service was unhealthy.


    7. Avoid False Positives from Past Data

    Make sure you're querying only recent data (e.g., 5m or 10m) in the range vector selectors.

    Also, avoid using offset unless necessary — especially if you're trying to compare current behavior only.


    8. (Optional) Alerting Rule Example

    For real-time alerting, define a recording rule or alerting rule:

    groups:
      - name: service-health-alerts
        rules:
          - record: service_unhealthy
            expr: |
              (
                rate(http_requests_total{status=~"5..|4.."}[5m]) > 0
                or
                avg_over_time(http_request_latency_seconds{quantile="0.99"}[5m]) > 1
                or
                rate(retries_total[5m]) > 0
                or
                absent(http_requests_total)
                or
                count_over_time(http_requests_total[5m]) == 0
              ) * 1
    
          - alert: ServiceUnhealthy
            expr: service_unhealthy == 1
            for: 2m
            labels:
              severity: warning
            annotations:
              summary: "Service {{ $labels.job }} is unhealthy"
              description: "Service {{ $labels.job }} has shown signs of failure (errors, latency, no data)"
    

    ✅ Summary

    FeatureSolution
    Combine charts into one viewUse PromQL to create a unified health metric per service
    Detect silent failuresUse absent() and count_over_time()
    Avoid old event false positivesLimit range vectors to short windows (e.g., [5m])
    Show in GrafanaUse Status History or Time Series panel with heatmaps
    Real-time alertingDefine alerting rules based on the same metric

    🛠 Tools & Plugins You May Need


    Would you like me to help write a full example dashboard JSON or alerting config tailored to your metrics?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.