Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Build a code generation quality gate that requires auto-generated diffs to pass contract…

  1. User

    Build a code generation quality gate that requires auto-generated diffs to pass contract tests, static analysis, and human review heuristics.

  2. GLM Assistant

    The user wants a "code generation quality gate". This implies a system or process that evaluates code generated by an AI or a template engine before it is committed or merged.

    The requirements for this gate are:

    1. Auto-generated diffs: The input is the difference between the existing code and the new code.
    2. Contract Tests: The code must adhere to a specific interface or behavior contract.
    3. Static Analysis: The code must pass linting, type checking, and complexity analysis.
    4. Human Review Heuristics: This is the tricky part. It suggests an automated check that simulates or predicts human review feedback (e.g., style, readability, potential bugs) or enforces guidelines that usually require human judgment (e.g., naming conventions, magic numbers, security anti-patterns).

    Approach:

    I need to design a Python-based system that acts as this gate. It should be modular so different "checkers" can be plugged in.

    Components:

    1. QualityGate class: The main orchestrator. It takes the diff and the full file context.
    2. Check abstract base class: Defines the interface for a check (e.g., run(diff, context) -> Result).
    3. Specific Checks:
      • ContractTestCheck: Runs specific unit tests or validates an API schema (like OpenAPI or gRPC) against the generated code. Self-correction: Actually running unit tests on a diff is hard without execution environment. I will simulate this by checking if the generated function matches a signature requirement or mocking the test run.
      • StaticAnalysisCheck: Uses tools like pylint, flake8, or mypy (conceptually). I'll implement a simplified version that checks for syntax errors or common bad patterns using AST.
      • HumanReviewHeuristicCheck: This needs to look for "smells". Long functions, poor variable names, lack of docstrings, hard-coded credentials.

    Refining the "Human Review Heuristics": Since I cannot actually ask a human in an automated gate, I will implement rules that catch things a human would catch:

    • Naming conventions (snake_case vs camelCase).
    • Function length.
    • Presence of comments/docstrings.
    • Security risks (e.g., eval(), hardcoded passwords).
    • Cyclomatic complexity (approximation).

    The Workflow:

    1. Input: original_code, generated_code.
    2. Compute diff.
    3. Pass through the Gate.
    4. Output: Pass/Fail with a detailed report.

    Implementation Details:

    • I'll use Python's ast module for static analysis and heuristics because it's built-in and robust.
    • I'll mock the "Contract Test" part because running actual binary tests requires a complex environment, but I'll structure the code so it could call a subprocess.
    • I need a Diff class or helper to extract the changed lines.

    Drafting the Code Structure:

    • Data Structures: CheckResult (status, message).
    • Analyzers:
      • StaticAnalyzer: Uses ast to check syntax.
      • ContractValidator: Mocking a schema check.
      • HeuristicValidator: Checks for complexity, naming, security.

    Specific Heuristic Rules to Implement:

    1. Naming: check if variable names are too short (< 2 chars) or "bad" names (foo, bar, temp) in production code.
    2. Size: check if the added function is too long.
    3. Security: check for eval, exec.
    4. Diff context: ensure the diff isn't too large (huge diffs are hard to review).

    Let's write the code.

    • Imports: ast, difflib, typing, dataclasses.
    • Main Class: CodeQualityGate.
    • Mock Data: I'll create a scenario where an LLM generates a Python function.

    Refining the Contract Test: The prompt asks for "Contract Tests". In a real scenario, this means "If I changed the add_user function, does it still return a User ID?". I will implement a InterfaceContractCheck that ensures the generated function signature matches an expected signature (e.g., defined in an abstract base class or a config file).

    Refining Static Analysis: I'll simulate mypy or pylint by just doing AST parsing to ensure it's valid Python code and perhaps checking for unused imports.

    Refining Human Review Heuristics: This is the "AI Reviewer".

    • Rule 1: No "magic numbers".
    • Rule 2: Docstrings required for public methods.
    • Rule 3: Variable naming (snake_case for Python).

    Putting it together:

    1. DiffParser: Extracts the new/changed blocks.
    2. ContractTestRunner: Checks if the AST matches a definition.
    3. StaticAnalyzer: Basic linting via AST.
    4. HeuristicScanner: Naming, complexity, security patterns.
    5. Gate: Runs all three, aggregates results.

    Self-Correction during thought process: The prompt asks to build the gate. I should provide a complete, runnable script that demonstrates the gate failing bad code and passing good code.

    I will structure the output as a Python library implementation + an example usage.

    • Input: A diff string (or before/after code).
    • Output: A JSON report or printed report.

    Let's add a "Mock LLM" step to show generation, then the Gate processing that generation.

    Detailed Logic for Heuristics:

    • Naming: Regex for snake_case.
    • Security: ast visitor looking for calls to dangerous functions.
    • Complexity: Counting if/for statements in the new node.

    Detailed Logic for Contract:

    • I'll assume a "Contract Registry" exists (a dictionary of expected function signatures). The gate checks if a modified function is in the registry and validates its args/return types (annotations).

    Detailed Logic for Static Analysis:

    • ast.parse. If it raises SyntaxError, fail immediately.

    Final Plan:

    1. Define CheckStatus (Pass/Fail/Warning).
    2. Define CheckResult.
    3. Define CodeQualityGate.
    4. Implement ContractTestCheck (validates signature).
    5. Implement StaticAnalysisCheck (validates syntax + basic style).
    6. Implement HeuristicCheck (complexity, naming, security).
    7. Demonstrate with a "Bad" generation and a "Good" generation.

    Let's write the Python code.

    • Heuristic detail: Look for TODO or FIXME in generated code. A common LLM failure mode.

    • Diff detail: Use unified_diff format for input to make it realistic, but internal processing is easier on the full "after" text. I will accept "after" text and "before" text (or just "after" for new files). Let's assume we have the full generated content and the original content.

    • Code Contract: I'll define a ServiceContract class that maps function names to expected arguments.

    • Static Analysis: Use ast to walk the tree.

    • Heuristics: Use ast to check Cyclomatic Complexity (simple version) and re for naming.

    The code should be clean, commented, and runnable.

    One specific thing: "Auto-generated diffs". This usually implies the input is a diff. Processing raw diffs to extract full function context is hard. Simplification: I will write a DiffApplier or simply assume the input to the gate is the resulting code (the "after" state), or provide a helper that applies the diff. Actually, a "Quality Gate" usually runs in a CI pipeline where you have the file state. I will assume the input is the proposed code to be committed.

    Step-by-step Code Construction:

    1. Setup: Imports and Data Classes.
    2. Contract Checker: Mocking a check against an expected schema (e.g., OpenAPI or Pydantic model).
    3. Static Analysis: Valid Python syntax check.
    4. Heuristics:
      • SecurityVisitor: detect eval, exec, hardcoded keys.
      • StyleVisitor: detect long lines (via source split), lack of docstrings on functions.
      • ComplexityVisitor: count branches.
    5. Main Gate: Aggregates results.
    6. Example: Define a "contract" (interface). Generate bad code -> Gate fails. Generate good code -> Gate passes.

    Let's refine the "Human Review Heuristics". It needs to catch things like:

    • "Variable x is too ambiguous." -> Check for 1-char variable names.
    • "Function process_data is too complex (CCN > 10)." -> Count if/while/for.
    • "Found hardcoded API key." -> Regex for key=... or password=....

    This looks solid. I will present the solution as a Python framework.

    One final touch: The "Gate" should allow configuration (e.g., max complexity = 10).

    Dependencies: None external (standard library only) to ensure it runs anywhere easily.

    Let's double check the "Contract Test" part. A contract test usually spins up the service and hits endpoints. In a code generation context (pre-commit), we can't do that easily without a heavy env. Alternative: "Schema Validation". Validate that the generated function signature matches the OpenAPI spec or a Pydantic model definition. I will implement this as "Signature Enforcement".

    Resulting Code Structure:

    class QualityGate:
        def __init__(self, config): ...
        def evaluate(self, generated_code): ...
    
    # Implementations of checks...
    

    I will add a diff helper to show what changed, as the prompt specifically mentions diffs. difflib.unified_diff is good for visualization.

    Let's build it. This solution implements a Code Generation Quality Gate framework in Python.

    This framework intercepts code generated by an AI or template engine and runs it through three distinct layers of verification before it is allowed to be written to disk or committed:

    1. Contract Tests: Verifies the code adheres to predefined interfaces (signatures and types).
    2. Static Analysis: Ensures the code is syntactically correct and follows linting rules (using Python's ast).
    3. Human Review Heuristics: Simulates a senior engineer's scrutiny by checking for "code smells," complexity, and security anti-patterns that linters often miss.

    The Implementation

    import ast
    import re
    import sys
    import json
    import typing
    from dataclasses import dataclass, field
    from enum import Enum
    from typing import List, Dict, Optional, Callable
    
    # ==========================================
    # 1. Data Structures & Configuration
    # ==========================================
    
    class Severity(Enum):
        CRITICAL = 1  # Blocks commit immediately
        WARNING = 2   # Warns user, requires override flag
        INFO = 3      # Suggestions
    
    @dataclass
    class Violation:
        rule_id: str
        message: str
        severity: Severity
        line_number: Optional[int] = None
    
    @dataclass
    class GateReport:
        passed: bool = True
        contract_violations: List[Violation] = field(default_factory=list)
        static_violations: List[Violation] = field(default_factory=list)
        heuristic_violations: List[Violation] = field(default_factory=list)
    
        def __str__(self):
            if self.passed:
                return "✅ QUALITY GATE PASSED"
            
            output = ["❌ QUALITY GATE FAILED\n"]
            
            categories = [
                ("Contract Tests", self.contract_violations),
                ("Static Analysis", self.static_violations),
                ("Review Heuristics", self.heuristic_violations)
            ]
    
            for cat_name, violations in categories:
                if violations:
                    output.append(f"--- {cat_name} ---")
                    for v in violations:
                        loc = f"Line {v.line_number}: " if v.line_number else ""
                        output.append(f"[{v.severity.name}] {loc}{v.rule_id}: {v.message}")
                    output.append("")
            
            return "\n".join(output)
    
    # ==========================================
    # 2. The Gate Components
    # ==========================================
    
    class ContractEnforcer:
        """
        Ensures the generated code fits into the existing system architecture.
        Validates function signatures against a schema.
        """
        def __init__(self, required_contracts: Dict[str, Dict]):
            # Schema format: {"function_name": {"args": ["arg1", "arg2"], "returns": "List[int]"}}
            self.contracts = required_contracts
    
        def validate(self, code: str) -> List[Violation]:
            violations = []
            try:
                tree = ast.parse(code)
                defined_funcs = {node.name: node for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)}
    
                for func_name, contract in self.contracts.items():
                    if func_name not in defined_funcs:
                        violations.append(Violation(
                            rule_id="CTR_001",
                            message=f"Missing required function definition: '{func_name}'",
                            severity=Severity.CRITICAL
                        ))
                        continue
    
                    # Check Argument Names (Simple check for demo)
                    func_node = defined_funcs[func_name]
                    actual_args = [arg.arg for arg in func_node.args.args]
                    expected_args = contract.get('args', [])
                    
                    if actual_args != expected_args:
                        violations.append(Violation(
                            rule_id="CTR_002",
                            message=f"Signature mismatch for '{func_name}'. Expected args {expected_args}, got {actual_args}",
                            severity=Severity.CRITICAL,
                            line_number=func_node.lineno
                        ))
            except Exception as e:
                 violations.append(Violation("CTR_ERR", f"Contract parsing failed: {str(e)}", Severity.CRITICAL))
            
            return violations
    
    class StaticAnalyzer:
        """
        Performs linting and syntax verification without executing code.
        """
        def validate(self, code: str) -> List[Violation]:
            violations = []
            
            # 1. Syntax Check
            try:
                tree = ast.parse(code)
            except SyntaxError as e:
                violations.append(Violation(
                    rule_id="SYN_001",
                    message=f"Syntax Error: {e.msg}",
                    severity=Severity.CRITICAL,
                    line_number=e.lineno
                ))
                return violations # Cannot proceed if syntax is broken
    
            # 2. Import Validation (Example:Disallow specific unsafe imports)
            forbidden_imports = {'pickle', 'subprocess', 'os.system'} # Simplification
            for node in ast.walk(tree):
                if isinstance(node, ast.Import):
                    for alias in node.names:
                        if alias.name in forbidden_imports:
                            violations.append(Violation(
                                rule_id="LINT_001",
                                message=f"Forbidden import '{alias.name}' detected.",
                                severity=Severity.CRITICAL,
                                line_number=node.lineno
                            ))
                if isinstance(node, ast.ImportFrom):
                    if node.module in forbidden_imports:
                         violations.append(Violation(
                            rule_id="LINT_001",
                            message=f"Forbidden import from '{node.module}' detected.",
                            severity=Severity.CRITICAL,
                            line_number=node.lineno
                        ))
            return violations
    
    class HumanReviewHeuristics:
        """
        Analyzes code for readability, maintainability, and 'smells' 
        that strictly typed linters might miss but humans would flag.
        """
        def validate(self, code: str) -> List[Violation]:
            violations = []
            
            try:
                tree = ast.parse(code)
            except SyntaxError:
                return [Violation("HEUR_ERR", "Cannot run heuristics on broken code", Severity.CRITICAL)]
    
            # 1. Complexity Check (Deep Nesting)
            for node in ast.walk(tree):
                if isinstance(node, (ast.If, ast.For, ast.While)):
                    depth = self._get_nesting_depth(node)
                    if depth > 3:
                        violations.append(Violation(
                            rule_id="HEUR_001",
                            message=f"Deep nesting detected (depth {depth}). Refactor for readability.",
                            severity=Severity.WARNING,
                            line_number=node.lineno
                        ))
    
            # 2. "Magic Numbers" Check
            for node in ast.walk(tree):
                 if isinstance(node, ast.Constant):
                     # Exclude common 0, 1, strings
                     if isinstance(node.value, (int, float)) and node.value not in [0, 1, -1, 2]:
                         violations.append(Violation(
                            rule_id="HEUR_002",
                            message=f"Magic number {node.value} found. Define as a constant.",
                            severity=Severity.INFO,
                            line_number=node.lineno
                        ))
            
            # 3. Naming Conventions (Auto-generated code often uses x, y, z or var_1)
            for node in ast.walk(tree):
                if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store):
                    if len(node.id) == 1 and node.id not in ['i', 'j', 'k', 'x', 'y', 'n']:
                        violations.append(Violation(
                            rule_id="HEUR_003",
                            message=f"Variable name '{node.id}' is too short/vague.",
                            severity=Severity.INFO,
                            line_number=node.lineno
                        ))
                    elif re.match(r"var_\d+", node.id):
                         violations.append(Violation(
                            rule_id="HEUR_004",
                            message=f"Variable '{node.id}' looks machine-generated. Rename descriptively.",
                            severity=Severity.WARNING,
                            line_number=node.lineno
                        ))
    
            return violations
    
        def _get_nesting_depth(self, node, current_depth=1):
            max_depth = current_depth
            for child in ast.iter_child_nodes(node):
                if isinstance(child, (ast.If, ast.For, ast.While)):
                    child_depth = self._get_nesting_depth(child, current_depth + 1)
                    if child_depth > max_depth:
                        max_depth = child_depth
            return max_depth
    
    # ==========================================
    # 3. The Quality Gate Orchestrator
    # ==========================================
    
    class CodeQualityGate:
        def __init__(self, required_contracts: Dict[str, Dict]):
            self.enforcer = ContractEnforcer(required_contracts)
            self.analyzer = StaticAnalyzer()
            self.heuristics = HumanReviewHeuristics()
    
        def process(self, generated_code: str) -> GateReport:
            report = GateReport()
    
            # Step 1: Static Analysis (Must pass first)
            report.static_violations = self.analyzer.validate(generated_code)
            
            # Step 2: Contract Tests
            report.contract_violations = self.enforcer.validate(generated_code)
            
            # Step 3: Heuristics
            report.heuristic_violations = self.heuristics.validate(generated_code)
    
            # Determine Pass/Fail
            # CRITICAL violations in any category block the code
            has_critical = False
            for v in report.static_violations + report.contract_violations + report.heuristic_violations:
                if v.severity == Severity.CRITICAL:
                    has_critical = True
                    break
            
            # WARNING level heuristics also block by default in strict mode, 
            # but we simulate a 'review required' flag here.
            # For this demo, CRITICAL fails it, WARNING fails it (strict).
            has_warning = any(v.severity == Severity.WARNING for v in report.heuristic_violations)
    
            if has_critical or has_warning:
                report.passed = False
                
            return report
    
    # ==========================================
    # 4. Usage Demonstration
    # ==========================================
    
    def demonstrate_gate():
        # A. Define the Contract (The expected interface)
        # We expect the AI to generate a function 'calculate_stats' taking 'data' and returning nothing (void in python implies None)
        contracts = {
            "calculate_stats": {
                "args": ["data", "threshold"], 
                "returns": "dict"
            }
        }
    
        # B. Initialize Gate
        gate = CodeQualityGate(contracts)
    
        # C. Scenario 1: Bad Code (Hallucinated signature, deep nesting, magic numbers)
        print("--- SCENARIO 1: BAD GENERATION ---")
        bad_code = """
    import pickle # Dangerous import
    
    def calculate_stats(data, wrong_arg): # Contract violation: signature mismatch
        x = 1
        if data: # Depth 1
            for row in data: # Depth 2
                if row > 0: # Depth 3
                    if row > 100: # Depth 4 (Too deep)
                        val_123 = 55 # Bad variable name & magic number
                        print("Too high")
        return {}
    """
        report = gate.process(bad_code)
        print(report)
        assert not report.passed
    
        # D. Scenario 2: Good Code
        print("\n--- SCENARIO 2: GOOD GENERATION ---")
        good_code = """
    import json
    
    def calculate_stats(data, threshold):
        result = {}
        for row in data:
            if row > threshold:
                result['high'] = result.get('high', 0) + 1
        return result
    """
        report = gate.process(good_code)
        print(report)
        assert report.passed
    
    if __name__ == "__main__":
        demonstrate_gate()
    

    Key Features

    1. ContractEnforcer (The Interface Guard):

      • Uses ast (Abstract Syntax Tree) to parse the code structure without executing it.
      • Compares the generated function signatures against a required_contracts dictionary.
      • Catches: Hallucinated function names, missing arguments, or incorrect return types (if type hints were enforced).
    2. StaticAnalyzer (The Linter):

      • Checks for valid Python syntax.
      • Scans for forbidden imports (e.g., pickle, subprocess) which are common security risks in AI-generated code.
      • Catches: Syntax errors, Security risks.
    3. HumanReviewHeuristics (The "Senior Dev" Sim):

      • Nesting Depth: Walks the AST to calculate cyclomatic complexity/nesting. AI often generates deeply nested if/else chains; this flags them.
      • Naming Smells: Uses Regex/AST to detect machine-generated variable names (like var_1, val_123) or single-letter variables that aren't loop indices.
      • Magic Numbers: Detects hardcoded numbers that should be constants.
      • Catches: Maintainability issues and code that looks "robotic."

    How to Integrate

    In a real pipeline, you would call CodeQualityGate(generated_code) immediately after the LLM returns the code block and before writing it to your repository. If report.passed is False, you reject the generation and trigger a retry or a human review task.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.