Skip to content
TheChatArchiveThe Chat Archive
GPT 5.1· openaiHugging Face Frontier Reasoning Traces

How do you approach requirements engineering for AI or machine learning systems, given…

  1. User

    How do you approach requirements engineering for AI or machine learning systems, given their unpredictable and emergent behavior?

  2. Assistant

    Considering requirements engineering for AI systems

    The user's question about approaching requirements engineering for AI and machine learning systems is definitely a complex one, especially due to their unpredictable nature. I want to provide a well-structured and reasonably detailed answer, considering principles and techniques that differ from traditional software requirements engineering. I'll also address uncertainty, non-determinism, data needs, and continuous requirements. Mentioning standards like ISO/IEC/IEEE 29148 would also be relevant. This will help frame the discussion effectively.Outlining requirements engineering for AI

    I should start by outlining challenges like non-determinism, dependence on data, concept drift, and issues relating to fairness and explanation. Then, I can propose an extended requirements engineering lifecycle. This includes early stakeholder analysis that focuses on value and risk assessment alongside defining use case boundaries and operational design domains. It’s important to categorize requirements into business goals, functional needs, performance metrics specific to machine learning, data requirements, non-functional aspects, and ethical compliance considerations. This will ensure a comprehensive approach.Approaching unpredictability in AI requirements

    I’m considering how to handle unpredictability by thinking of requirements in terms of distributions and confidence intervals instead of fixed points. Acceptance criteria would involve performance metrics from test sets and validation processes. I might introduce concepts like “requirements envelopes” for flexibility and utilize scenario-based approaches to cover worst-case scenarios. Documenting assumptions about data and user behavior is crucial. I also want to emphasize continuous requirements engineering with ongoing monitoring, feedback loops, and using model cards for clarity. Contracts for AI components should incorporate statistical reasoning for better predictability.Addressing safety in ML scenarios

    In safety-critical scenarios, I want to draw analogies to standards like ISO 26262 and DO-178C. I plan on utilizing hazard analysis methods like HAZOP and STPA tailored for machine learning, incorporating risk-based safety margins and partial formal verification when applicable. My methodology will include framing the problem and conducting stakeholder analysis, defining success metrics, and collaborating with data scientists and experts to elicit requirements. I’ll document trade-offs, develop a verification plan, and establish monitoring requirements to ensure safety while also testing across various scenarios to manage emergent behavior.Structuring requirements engineering for ML systems

    I'm considering using a rule-based wrapper around machine learning to enforce invariants. The emergence of unpredictable properties means we can still define "unacceptable behaviors" and implement guardrails and alignment strategies. I want to maintain conciseness while being detailed. I’ll structure the document to address the differences between traditional requirements engineering and ML, outline a clear approach, discuss specific techniques for unpredictability, offer a short example, and provide a bullet list of requirement categories with sample phrasing. This will guide my writing in a coherent and thorough way. I treat requirements engineering for AI/ML as risk‑driven, statistical, and continuous, rather than “specify once, implement, done.” You can’t fully specify internal behavior, but you can specify:

    • What the system is allowed and expected to do at its boundaries
    • How well it must perform, in which conditions, with what level of confidence
    • What harms must be prevented, and with what safeguards and monitoring

    Below is a structured way to approach it.


    1. Start with problem framing and risk

    Before talking about models, make the decision explicit.

    • Decision and outcome

      • What decision will the AI influence? (e.g., “approve loan,” “flag transaction,” “propose diagnosis”)
      • What is the business or mission objective (e.g., reduce manual review by 40% with no more than 1% additional false positives)?
    • Stakeholders and harms

      • Who can be helped or harmed (end users, subjects of decisions, regulators, ops teams)?
      • What are plausible harm scenarios (bias, unsafe actions, privacy loss, lost revenue, reputational damage)?
    • Criticality / risk level

      • Map the use case to a risk category (e.g. EU AI Act style: minimal, limited, high‑risk, prohibited).
      • The higher the risk, the more stringent the requirements (performance, audits, human oversight).

    This gives you the requirements “tone”: error tolerance, necessary explainability, and control mechanisms.


    2. Define system boundaries and human/AI roles

    ML components almost never stand alone. Clarifying the surrounding system lets you bound “unpredictable” behavior.

    • System context and interfaces

      • Inputs: where the data comes from, at what frequency, with what quality guarantees.
      • Outputs: who/what consumes predictions, in what format, under what timing constraints.
    • Human–AI interaction

      • Human-in-the-loop: AI provides a recommendation; human makes final decision.
      • Human-on-the-loop: AI makes decisions; human monitors and can override.
      • Fully automated: AI decisions execute directly on the environment.
    • Operational Design Domain (ODD)

      • Explicitly define where the model is allowed to operate:
        • e.g. “Adults 18–65, US only, credit history ≥ 12 months, loan amount < $X.”
      • Outside the ODD: define fallback behavior (e.g., route to human, deny, or use simple rule-based logic).

    These become hard constraints and high-level functional requirements for the system.


    3. Classify requirements: not just “accuracy”

    Think in layers. For AI systems I typically group requirements into:

    3.1. Business / outcome requirements

    Examples:

    • “Reduce average handling time for support tickets by 25% while maintaining CSAT ≥ 4.3/5.”
    • “Detect at least 90% of fraudulent transactions, adding no more than 0.5% false positives compared to current system.”

    These anchor the design trade-offs.

    3.2. System-level functional requirements

    These are still traditional, deterministic requirements around the ML:

    • “For each submitted transaction, the system shall produce a risk score in [0,1] within 200 ms.”
    • “If the fraud score exceeds 0.9, the transaction shall be held for manual review.”
    • “If the model reports uncertainty above U, the system shall escalate to a human reviewer.”

    They describe what the system does at observable interfaces, not how the model decides internally.

    3.3. ML performance requirements (statistical)

    Here you accept that behavior is probabilistic and specify targets with confidence and scope:

    • Overall metrics:

      • “On a holdout set representative of production traffic in region X, the model shall achieve:
        • ROC-AUC ≥ 0.93,
        • Precision@k ≥ 0.85 (k = top 5% of scores),
          with 95% confidence.”
    • Segment-specific metrics (to avoid “works on average, fails on subgroups”):

      • “Recall (sensitivity) shall be ≥ 0.80 for each age group: 18–30, 31–50, >50.”
      • “False positive rate difference between protected groups A and B shall be < 0.03.”
    • Thresholds and “performance envelopes”:

      • Not one number, but acceptable ranges: “F1 score shall be in [0.84, 0.90]; if below 0.84, block deployment; if below 0.87, flag for improvement.”

    You’re specifying distributions and confidence, not deterministic correctness for every instance.

    3.4. Data requirements

    Because behavior emerges from data, requirements on data are as important as requirements on code:

    • Coverage and representativeness

      • “Training data shall cover the past 24 months and all operating regions where the system is deployed.”
      • “For each key segment (by age, gender, geography), the training set shall contain at least N labeled examples.”
    • Label quality

      • “Labeling guidelines shall be documented and training provided; inter-annotator agreement κ ≥ 0.8 on a validation subset.”
    • Provenance, privacy, compliance

      • “All PII in training data shall be anonymized/pseudonymized per GDPR; data sources and transformations shall be documented (datasheet for dataset).”

    3.5. Non-functional requirements (quality attributes)

    For ML, I explicitly add:

    • Robustness & generalization

      • “Accuracy under realistic input noise or missingness patterns shall not drop by more than 3 percentage points.”
      • “The model shall detect and flag out-of-distribution inputs with at least 95% recall on a curated OOD test set.”
    • Fairness / non-discrimination

      • Choose relevant fairness notions and turn them into numeric constraints.
        E.g.: “Difference in approval rate between groups A and B, after controlling for income and credit history, shall be ≤ 5%.”
    • Explainability & transparency

      • “For any automated denial, the system shall provide a user-facing explanation citing at least 2–3 most influential factors, in non-technical language.”
      • “Feature importance and example-based explanations shall be auditable and reproducible.”
    • Performance & scalability

      • “99th percentile latency ≤ 500 ms for 1000 concurrent requests.”
    • Security & abuse resistance

      • “The system shall log and rate-limit inputs indicative of prompt injection or adversarial probing.”
      • “Adversarial perturbations below L2 distance ε shall not change classification for > X% of test samples” (where applicable).
    • Auditability & traceability

      • “For each prediction, the system shall log: model version, data hash, key features, and downstream decision taken.”

    4. Handling unpredictability and emergent behavior

    You cannot eliminate unpredictability, but you can contain and manage it.

    4.1. Use safe boundaries and guardrails

    Define hard constraints that must never be violated, independent of the model’s opinion:

    • “The dosage recommendation must never exceed the maximum allowed by clinical guidelines.”
    • “The model may never directly execute financial transfers above $X without human confirmation.”

    Often this means wrapping the ML with rule-based logic or constraints that enforce domain invariants.

    4.2. Abstain and escalate on uncertainty

    Require the model to know when it doesn’t know:

    • “The model shall output calibrated probabilities (Brier score ≤ B on validation).”
    • “If prediction confidence < 0.7 or input flagged as OOD, the system shall abstain and send to human review.”
    • “Not more than 20% of total cases should be abstentions under normal operating conditions.”

    This turns unpredictable internal behavior into predictable system-level behavior: “If unsure → ask a human.”

    4.3. Scenario-based and stress testing

    Instead of attempting to specify behavior for all inputs, specify behavior for classes of scenarios:

    • Normal cases: requirements on aggregate metrics.
    • Edge cases: explicitly defined scenarios (e.g., rare but critical conditions) with stronger constraints.
    • Adversarial/misuse cases: “The system shall not generate content that matches any of the prohibited patterns A, B, C.”

    Then require test coverage:

    • “The test suite shall include at least N scenarios per identified hazard category.”
    • “All red-team prompts from the curated list shall be blocked or deflected.”

    4.4. Treat models as hypotheses; requirements co-evolve

    With ML, feasibility is often unknown upfront. I usually plan for requirements–model co-design:

    • Start with provisional performance targets based on prior experience or baseline.
    • Build baseline models quickly to check whether goals are even in range.
    • If not, either:
      • Improve data/model, or
      • Relax requirements, or
      • Change the problem framing (e.g., assist human instead of fully automate).

    This loop should be explicit in the RE process, not treated as failure.


    5. Continuous requirements and monitoring in production

    Because behavior can drift, requirements don’t end at deployment.

    Define operational requirements like:

    • Monitoring

      • “The system shall track and display daily: key performance metrics (e.g., precision, recall), data drift statistics, fairness metrics for groups A/B/C.”
      • “If PSI (Population Stability Index) on key features exceeds 0.2, alert the on-call engineer.”
    • Retraining / model update policies

      • “Models shall be retrained at least every 3 months or when performance drops > 2 percentage points on the production validation slice.”
      • “Any new model version must pass a predefined regression test suite and fairness checks before rollout.”
    • Governance

      • “A model change impacting high-risk decisions requires sign-off from data science lead, product owner, and compliance representative.”
      • “All model and data changes shall be documented via model cards and dataset datasheets.”

    Think of this as continuous requirements engineering: discovery in production feeds back into updated requirements.


    6. Concrete mini-example

    Suppose you’re building an AI triage assistant for an emergency department.

    Some key requirements might look like:

    • Scope / ODD

      • “System is only used for adult patients (>18) in non–cardiac-arrest situations; pediatric and cardiac arrest cases always bypass to human triage nurse.”
    • Performance

      • “For identifying high-acuity cases, model recall ≥ 0.95 and precision ≥ 0.80 on a test set representative of hospital’s patient mix, 95% CI.”
      • “Recall shall be ≥ 0.93 for each subgroup by sex and major ethnicity categories.”
    • Safety and guardrails

      • “System recommendations are advisory; triage nurse retains ultimate authority and must confirm or override.”
      • “System shall never recommend discharge for patients with red-flag symptoms list R (chest pain + hypotension, etc.), regardless of model score.”
    • Explainability

      • “For each triage level suggestion, system displays top 3 clinical factors and links to the relevant guideline section.”
    • Monitoring

      • “Weekly audit compares AI-recommended triage vs actual outcomes (ICU admission, mortality); any degradation triggers review.”

    The model’s internal logic is complex and partly opaque, but the system’s behavior is constrained, auditable, and measurable.


    7. Summary

    To handle AI/ML’s emergent and unpredictable behavior in requirements engineering:

    1. Frame the decision problem and risks first, not the model.
    2. Define clear system boundaries, human/AI roles, and an operational design domain.
    3. Specify multi-layered requirements:
      • Business outcomes
      • System-level functions
      • Statistical performance targets
      • Data quality/coverage
      • Non-functional properties like robustness, fairness, explainability, auditability
    4. Accept and encode uncertainty:
      • Ranges, distributions, confidence levels, and abstention behavior.
    5. Use guardrails, scenario-based testing, and human oversight to contain emergent behavior.
    6. Treat requirements as living, with explicit monitoring, governance, and feedback loops from production.

    If you share a specific type of AI system (e.g., recommender, LLM assistant, medical classifier), I can sketch a tailored set of requirement categories and example statements for that context.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.