Build a code generation quality gate that requires auto-generated diffs to pass contract…
- User
Build a code generation quality gate that requires auto-generated diffs to pass contract tests, static analysis, and human review heuristics.
Assistant
The user wants a "code generation quality gate". This implies a system or process that evaluates code generated by an AI or a template engine before it is committed or merged.
The requirements for this gate are:
- Auto-generated diffs: The input is the difference between the existing code and the new code.
- Contract Tests: The code must adhere to a specific interface or behavior contract.
- Static Analysis: The code must pass linting, type checking, and complexity analysis.
- Human Review Heuristics: This is the tricky part. It suggests an automated check that simulates or predicts human review feedback (e.g., style, readability, potential bugs) or enforces guidelines that usually require human judgment (e.g., naming conventions, magic numbers, security anti-patterns).
Approach:
I need to design a Python-based system that acts as this gate. It should be modular so different "checkers" can be plugged in.
Components:
QualityGateclass: The main orchestrator. It takes the diff and the full file context.Checkabstract base class: Defines the interface for a check (e.g.,run(diff, context) -> Result).- Specific Checks:
ContractTestCheck: Runs specific unit tests or validates an API schema (like OpenAPI or gRPC) against the generated code. Self-correction: Actually running unit tests on a diff is hard without execution environment. I will simulate this by checking if the generated function matches a signature requirement or mocking the test run.StaticAnalysisCheck: Uses tools likepylint,flake8, ormypy(conceptually). I'll implement a simplified version that checks for syntax errors or common bad patterns using AST.HumanReviewHeuristicCheck: This needs to look for "smells". Long functions, poor variable names, lack of docstrings, hard-coded credentials.
Refining the "Human Review Heuristics": Since I cannot actually ask a human in an automated gate, I will implement rules that catch things a human would catch:
- Naming conventions (snake_case vs camelCase).
- Function length.
- Presence of comments/docstrings.
- Security risks (e.g.,
eval(), hardcoded passwords). - Cyclomatic complexity (approximation).
The Workflow:
- Input:
original_code,generated_code. - Compute
diff. - Pass through the Gate.
- Output: Pass/Fail with a detailed report.
Implementation Details:
- I'll use Python's
astmodule for static analysis and heuristics because it's built-in and robust. - I'll mock the "Contract Test" part because running actual binary tests requires a complex environment, but I'll structure the code so it could call a subprocess.
- I need a
Diffclass or helper to extract the changed lines.
Drafting the Code Structure:
- Data Structures:
CheckResult(status, message). - Analyzers:
StaticAnalyzer: Usesastto check syntax.ContractValidator: Mocking a schema check.HeuristicValidator: Checks for complexity, naming, security.
Specific Heuristic Rules to Implement:
- Naming: check if variable names are too short (< 2 chars) or "bad" names (foo, bar, temp) in production code.
- Size: check if the added function is too long.
- Security: check for
eval,exec. - Diff context: ensure the diff isn't too large (huge diffs are hard to review).
Let's write the code.
- Imports:
ast,difflib,typing,dataclasses. - Main Class:
CodeQualityGate. - Mock Data: I'll create a scenario where an LLM generates a Python function.
Refining the Contract Test: The prompt asks for "Contract Tests". In a real scenario, this means "If I changed the
add_userfunction, does it still return a User ID?". I will implement aInterfaceContractCheckthat ensures the generated function signature matches an expected signature (e.g., defined in an abstract base class or a config file).Refining Static Analysis: I'll simulate
mypyorpylintby just doing AST parsing to ensure it's valid Python code and perhaps checking for unused imports.Refining Human Review Heuristics: This is the "AI Reviewer".
- Rule 1: No "magic numbers".
- Rule 2: Docstrings required for public methods.
- Rule 3: Variable naming (snake_case for Python).
Putting it together:
DiffParser: Extracts the new/changed blocks.ContractTestRunner: Checks if the AST matches a definition.StaticAnalyzer: Basic linting via AST.HeuristicScanner: Naming, complexity, security patterns.Gate: Runs all three, aggregates results.
Self-Correction during thought process: The prompt asks to build the gate. I should provide a complete, runnable script that demonstrates the gate failing bad code and passing good code.
I will structure the output as a Python library implementation + an example usage.
- Input: A diff string (or before/after code).
- Output: A JSON report or printed report.
Let's add a "Mock LLM" step to show generation, then the Gate processing that generation.
Detailed Logic for Heuristics:
- Naming: Regex for snake_case.
- Security:
astvisitor looking for calls to dangerous functions. - Complexity: Counting
if/forstatements in the new node.
Detailed Logic for Contract:
- I'll assume a "Contract Registry" exists (a dictionary of expected function signatures). The gate checks if a modified function is in the registry and validates its args/return types (annotations).
Detailed Logic for Static Analysis:
ast.parse. If it raisesSyntaxError, fail immediately.
Final Plan:
- Define
CheckStatus(Pass/Fail/Warning). - Define
CheckResult. - Define
CodeQualityGate. - Implement
ContractTestCheck(validates signature). - Implement
StaticAnalysisCheck(validates syntax + basic style). - Implement
HeuristicCheck(complexity, naming, security). - Demonstrate with a "Bad" generation and a "Good" generation.
Let's write the Python code.
-
Heuristic detail: Look for
TODOorFIXMEin generated code. A common LLM failure mode. -
Diff detail: Use
unified_diffformat for input to make it realistic, but internal processing is easier on the full "after" text. I will accept "after" text and "before" text (or just "after" for new files). Let's assume we have the full generated content and the original content. -
Code Contract: I'll define a
ServiceContractclass that maps function names to expected arguments. -
Static Analysis: Use
astto walk the tree. -
Heuristics: Use
astto check Cyclomatic Complexity (simple version) andrefor naming.
The code should be clean, commented, and runnable.
One specific thing: "Auto-generated diffs". This usually implies the input is a diff. Processing raw diffs to extract full function context is hard. Simplification: I will write a
DiffApplieror simply assume the input to the gate is the resulting code (the "after" state), or provide a helper that applies the diff. Actually, a "Quality Gate" usually runs in a CI pipeline where you have the file state. I will assume the input is the proposed code to be committed.Step-by-step Code Construction:
- Setup: Imports and Data Classes.
- Contract Checker: Mocking a check against an expected schema (e.g., OpenAPI or Pydantic model).
- Static Analysis: Valid Python syntax check.
- Heuristics:
SecurityVisitor: detecteval,exec, hardcoded keys.StyleVisitor: detect long lines (via source split), lack of docstrings on functions.ComplexityVisitor: count branches.
- Main Gate: Aggregates results.
- Example: Define a "contract" (interface). Generate bad code -> Gate fails. Generate good code -> Gate passes.
Let's refine the "Human Review Heuristics". It needs to catch things like:
- "Variable
xis too ambiguous." -> Check for 1-char variable names. - "Function
process_datais too complex (CCN > 10)." -> Countif/while/for. - "Found hardcoded API key." -> Regex for
key=...orpassword=....
This looks solid. I will present the solution as a Python framework.
One final touch: The "Gate" should allow configuration (e.g., max complexity = 10).
Dependencies: None external (standard library only) to ensure it runs anywhere easily.
Let's double check the "Contract Test" part. A contract test usually spins up the service and hits endpoints. In a code generation context (pre-commit), we can't do that easily without a heavy env. Alternative: "Schema Validation". Validate that the generated function signature matches the OpenAPI spec or a Pydantic model definition. I will implement this as "Signature Enforcement".
Resulting Code Structure:
class QualityGate: def __init__(self, config): ... def evaluate(self, generated_code): ... # Implementations of checks...I will add a
diffhelper to show what changed, as the prompt specifically mentions diffs.difflib.unified_diffis good for visualization.Let's build it. This solution implements a Code Generation Quality Gate framework in Python.
This framework intercepts code generated by an AI or template engine and runs it through three distinct layers of verification before it is allowed to be written to disk or committed:
- Contract Tests: Verifies the code adheres to predefined interfaces (signatures and types).
- Static Analysis: Ensures the code is syntactically correct and follows linting rules (using Python's
ast). - Human Review Heuristics: Simulates a senior engineer's scrutiny by checking for "code smells," complexity, and security anti-patterns that linters often miss.
The Implementation
import ast import re import sys import json import typing from dataclasses import dataclass, field from enum import Enum from typing import List, Dict, Optional, Callable # ========================================== # 1. Data Structures & Configuration # ========================================== class Severity(Enum): CRITICAL = 1 # Blocks commit immediately WARNING = 2 # Warns user, requires override flag INFO = 3 # Suggestions @dataclass class Violation: rule_id: str message: str severity: Severity line_number: Optional[int] = None @dataclass class GateReport: passed: bool = True contract_violations: List[Violation] = field(default_factory=list) static_violations: List[Violation] = field(default_factory=list) heuristic_violations: List[Violation] = field(default_factory=list) def __str__(self): if self.passed: return "✅ QUALITY GATE PASSED" output = ["❌ QUALITY GATE FAILED\n"] categories = [ ("Contract Tests", self.contract_violations), ("Static Analysis", self.static_violations), ("Review Heuristics", self.heuristic_violations) ] for cat_name, violations in categories: if violations: output.append(f"--- {cat_name} ---") for v in violations: loc = f"Line {v.line_number}: " if v.line_number else "" output.append(f"[{v.severity.name}] {loc}{v.rule_id}: {v.message}") output.append("") return "\n".join(output) # ========================================== # 2. The Gate Components # ========================================== class ContractEnforcer: """ Ensures the generated code fits into the existing system architecture. Validates function signatures against a schema. """ def __init__(self, required_contracts: Dict[str, Dict]): # Schema format: {"function_name": {"args": ["arg1", "arg2"], "returns": "List[int]"}} self.contracts = required_contracts def validate(self, code: str) -> List[Violation]: violations = [] try: tree = ast.parse(code) defined_funcs = {node.name: node for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)} for func_name, contract in self.contracts.items(): if func_name not in defined_funcs: violations.append(Violation( rule_id="CTR_001", message=f"Missing required function definition: '{func_name}'", severity=Severity.CRITICAL )) continue # Check Argument Names (Simple check for demo) func_node = defined_funcs[func_name] actual_args = [arg.arg for arg in func_node.args.args] expected_args = contract.get('args', []) if actual_args != expected_args: violations.append(Violation( rule_id="CTR_002", message=f"Signature mismatch for '{func_name}'. Expected args {expected_args}, got {actual_args}", severity=Severity.CRITICAL, line_number=func_node.lineno )) except Exception as e: violations.append(Violation("CTR_ERR", f"Contract parsing failed: {str(e)}", Severity.CRITICAL)) return violations class StaticAnalyzer: """ Performs linting and syntax verification without executing code. """ def validate(self, code: str) -> List[Violation]: violations = [] # 1. Syntax Check try: tree = ast.parse(code) except SyntaxError as e: violations.append(Violation( rule_id="SYN_001", message=f"Syntax Error: {e.msg}", severity=Severity.CRITICAL, line_number=e.lineno )) return violations # Cannot proceed if syntax is broken # 2. Import Validation (Example:Disallow specific unsafe imports) forbidden_imports = {'pickle', 'subprocess', 'os.system'} # Simplification for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: if alias.name in forbidden_imports: violations.append(Violation( rule_id="LINT_001", message=f"Forbidden import '{alias.name}' detected.", severity=Severity.CRITICAL, line_number=node.lineno )) if isinstance(node, ast.ImportFrom): if node.module in forbidden_imports: violations.append(Violation( rule_id="LINT_001", message=f"Forbidden import from '{node.module}' detected.", severity=Severity.CRITICAL, line_number=node.lineno )) return violations class HumanReviewHeuristics: """ Analyzes code for readability, maintainability, and 'smells' that strictly typed linters might miss but humans would flag. """ def validate(self, code: str) -> List[Violation]: violations = [] try: tree = ast.parse(code) except SyntaxError: return [Violation("HEUR_ERR", "Cannot run heuristics on broken code", Severity.CRITICAL)] # 1. Complexity Check (Deep Nesting) for node in ast.walk(tree): if isinstance(node, (ast.If, ast.For, ast.While)): depth = self._get_nesting_depth(node) if depth > 3: violations.append(Violation( rule_id="HEUR_001", message=f"Deep nesting detected (depth {depth}). Refactor for readability.", severity=Severity.WARNING, line_number=node.lineno )) # 2. "Magic Numbers" Check for node in ast.walk(tree): if isinstance(node, ast.Constant): # Exclude common 0, 1, strings if isinstance(node.value, (int, float)) and node.value not in [0, 1, -1, 2]: violations.append(Violation( rule_id="HEUR_002", message=f"Magic number {node.value} found. Define as a constant.", severity=Severity.INFO, line_number=node.lineno )) # 3. Naming Conventions (Auto-generated code often uses x, y, z or var_1) for node in ast.walk(tree): if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store): if len(node.id) == 1 and node.id not in ['i', 'j', 'k', 'x', 'y', 'n']: violations.append(Violation( rule_id="HEUR_003", message=f"Variable name '{node.id}' is too short/vague.", severity=Severity.INFO, line_number=node.lineno )) elif re.match(r"var_\d+", node.id): violations.append(Violation( rule_id="HEUR_004", message=f"Variable '{node.id}' looks machine-generated. Rename descriptively.", severity=Severity.WARNING, line_number=node.lineno )) return violations def _get_nesting_depth(self, node, current_depth=1): max_depth = current_depth for child in ast.iter_child_nodes(node): if isinstance(child, (ast.If, ast.For, ast.While)): child_depth = self._get_nesting_depth(child, current_depth + 1) if child_depth > max_depth: max_depth = child_depth return max_depth # ========================================== # 3. The Quality Gate Orchestrator # ========================================== class CodeQualityGate: def __init__(self, required_contracts: Dict[str, Dict]): self.enforcer = ContractEnforcer(required_contracts) self.analyzer = StaticAnalyzer() self.heuristics = HumanReviewHeuristics() def process(self, generated_code: str) -> GateReport: report = GateReport() # Step 1: Static Analysis (Must pass first) report.static_violations = self.analyzer.validate(generated_code) # Step 2: Contract Tests report.contract_violations = self.enforcer.validate(generated_code) # Step 3: Heuristics report.heuristic_violations = self.heuristics.validate(generated_code) # Determine Pass/Fail # CRITICAL violations in any category block the code has_critical = False for v in report.static_violations + report.contract_violations + report.heuristic_violations: if v.severity == Severity.CRITICAL: has_critical = True break # WARNING level heuristics also block by default in strict mode, # but we simulate a 'review required' flag here. # For this demo, CRITICAL fails it, WARNING fails it (strict). has_warning = any(v.severity == Severity.WARNING for v in report.heuristic_violations) if has_critical or has_warning: report.passed = False return report # ========================================== # 4. Usage Demonstration # ========================================== def demonstrate_gate(): # A. Define the Contract (The expected interface) # We expect the AI to generate a function 'calculate_stats' taking 'data' and returning nothing (void in python implies None) contracts = { "calculate_stats": { "args": ["data", "threshold"], "returns": "dict" } } # B. Initialize Gate gate = CodeQualityGate(contracts) # C. Scenario 1: Bad Code (Hallucinated signature, deep nesting, magic numbers) print("--- SCENARIO 1: BAD GENERATION ---") bad_code = """ import pickle # Dangerous import def calculate_stats(data, wrong_arg): # Contract violation: signature mismatch x = 1 if data: # Depth 1 for row in data: # Depth 2 if row > 0: # Depth 3 if row > 100: # Depth 4 (Too deep) val_123 = 55 # Bad variable name & magic number print("Too high") return {} """ report = gate.process(bad_code) print(report) assert not report.passed # D. Scenario 2: Good Code print("\n--- SCENARIO 2: GOOD GENERATION ---") good_code = """ import json def calculate_stats(data, threshold): result = {} for row in data: if row > threshold: result['high'] = result.get('high', 0) + 1 return result """ report = gate.process(good_code) print(report) assert report.passed if __name__ == "__main__": demonstrate_gate()Key Features
-
ContractEnforcer (The Interface Guard):
- Uses
ast(Abstract Syntax Tree) to parse the code structure without executing it. - Compares the generated function signatures against a
required_contractsdictionary. - Catches: Hallucinated function names, missing arguments, or incorrect return types (if type hints were enforced).
- Uses
-
StaticAnalyzer (The Linter):
- Checks for valid Python syntax.
- Scans for forbidden imports (e.g.,
pickle,subprocess) which are common security risks in AI-generated code. - Catches: Syntax errors, Security risks.
-
HumanReviewHeuristics (The "Senior Dev" Sim):
- Nesting Depth: Walks the AST to calculate cyclomatic complexity/nesting. AI often generates deeply nested
if/elsechains; this flags them. - Naming Smells: Uses Regex/AST to detect machine-generated variable names (like
var_1,val_123) or single-letter variables that aren't loop indices. - Magic Numbers: Detects hardcoded numbers that should be constants.
- Catches: Maintainability issues and code that looks "robotic."
- Nesting Depth: Walks the AST to calculate cyclomatic complexity/nesting. AI often generates deeply nested
How to Integrate
In a real pipeline, you would call
CodeQualityGate(generated_code)immediately after the LLM returns the code block and before writing it to your repository. Ifreport.passedisFalse, you reject the generation and trigger a retry or a human review task.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.