Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-2024-07-18· OpenAIAi2 WildChat-4.8M (AI2)

uPDATE THE BELOW QUESTION THAT NEEDS TO BE ADDED IN JIRA Inputs required to identify the…

  1. User

    uPDATE THE BELOW QUESTION THAT NEEDS TO BE ADDED IN JIRA

    Inputs required to identify the performance metrics for model monitoring

    Requirement: There shall be provision to setup monitoring mechanisms in MLOps pipeline to track the model performance.

    The system shall integrate an observability tool to monitor the performance of deployed models.

    Could you clarify the following points to identify strategy to achieve this requirement: 1.What all are the current metrics used by UH team for model monitoring? 2.We we able to achieve some general performance metrics for model monitoring Based on our understanding we would require below items as well for better model monitoring 1.Data Drift (Percent)  2. Model Drift (Percent) Could you please confirm if these concepts are currently used by UH team?   Also, Ground truth data is required to calculate Model Accuracy & Performance Monitoring metrics such as Accuracy (Percent) → Compare predictions vs actual labels Precision (Percent) → True positives / (True positives + False positives) Recall (Percent) → True positives / (True positives + False negatives) F1-Score (Percent) → 2 × (Precision × Recall) / (Precision + Recall) False Positive Rate (Percent) → Model making wrong detections Could you please provide some more information on how these data are achieved in UH side?

  2. Assistant

    Here's an updated version of the question to be added to JIRA:


    Title: Inputs Required for Identifying Performance Metrics for Model Monitoring

    Requirement: We need to establish provisions for setting up robust monitoring mechanisms within the MLOps pipeline to effectively track the performance of deployed models. This includes the integration of an observability tool to enable real-time performance monitoring.

    Clarifications Needed: To devise a strategy that meets these requirements, could you please clarify the following points?

    1. Current Metrics: What performance metrics are currently utilized by the UH team for model monitoring?

    2. General Performance Metrics: Have we successfully established any general performance metrics suitable for model monitoring? Based on our understanding, we believe the following items are necessary for optimal monitoring:

      • Data Drift (Percentage)
      • Model Drift (Percentage)

      Could you confirm if the UH team currently employs these concepts?

    3. Ground Truth Data: Ground truth data is essential for calculating model accuracy and monitoring performance metrics. The following metrics are particularly significant:

      • Accuracy (Percentage): Comparison of predictions versus actual labels.
      • Precision (Percentage): True Positives / (True Positives + False Positives).
      • Recall (Percentage): True Positives / (True Positives + False Negatives).
      • F1-Score (Percentage): 2 × (Precision × Recall) / (Precision + Recall).
      • False Positive Rate (Percentage): The rate at which the model makes incorrect detections.

    Could you elaborate on how these data points are currently collected and utilized by the UH team?

    Additional Considerations: Any further insights into best practices or tools currently in use for model performance monitoring would also be greatly appreciated.


    This revision organizes the information clearly and adds a section for additional insights, making it comprehensive for the JIRA entry.