Compose a generative test case synthesizer leveraging LLMs plus static analysis: prompt…
- User
Compose a generative test case synthesizer leveraging LLMs plus static analysis: prompt design, verification, deduplication, and developer workflow.
- Assistant
Generative Test Case Synthesizer: LLM + Static Analysis
Executive Summary
The Generative Test Case Synthesizer (GTCS) is an automated tool that leverages Large Language Models (LLMs) augmented by static code analysis to generate high-quality, diverse unit test cases for software functions. It addresses key pain points in test generation: coverage gaps, manual effort, and redundancy. By combining structural insights from static analysis with the creative reasoning of LLMs, GTCS produces executable tests that pass verification, are deduplicated, and integrate seamlessly into developer workflows.
Key features:
- Static Analysis: Extracts code structure (e.g., params, branches, edge cases).
- Prompt Design: LLM-optimized templates for targeted test generation.
- Verification: Compiles, runs, and measures coverage of generated tests.
- Deduplication: Semantic and syntactic clustering to eliminate duplicates.
- Developer Workflow: CLI, IDE plugin, and CI/CD hooks for frictionless adoption.
GTCS supports languages like Python, JavaScript/TypeScript, Java, and Go, with extensible analyzers.
Architecture Overview
[Source Code] --> [Static Analyzer] --> [Prompt Engineer] --> [LLM Generator] --> [Raw Tests] | v [Verifier] <--> [Raw Tests] --> [Deduplicator] --> [Curated Tests] --> [Developer Workflow]- Static Analyzer: Parses code to produce a Code Insight Graph (CIG) – a structured JSON/YAML representation.
- Prompt Engineer: Crafts dynamic prompts using CIG data.
- LLM Generator: Calls an LLM (e.g., GPT-4o, Claude 3.5, or local Llama) to synthesize tests.
- Verifier: Validates tests for syntax, executability, and effectiveness.
- Deduplicator: Filters uniques using embeddings + heuristics.
- Developer Workflow: Outputs to files, IDE, or CI reports.
1. Static Analysis Module
Uses language-specific tools (e.g., Tree-sitter, Pyright, Javaparser) to build the CIG, capturing:
- Function signatures (params, return types, docs).
- Control flow (branches, loops, conditions).
- Data flows (dependencies, globals).
- Edge cases (nulls, boundaries, exceptions).
- Example CIG snippet (Python):
{ "function": "def divide(a: float, b: float) -> float", "params": [{"name": "a", "type": "float"}, {"name": "b", "type": "float"}], "branches": ["if b == 0: raise ZeroDivisionError"], "edges": ["b=0", "a=0", "inf", "nan"], "coverage_goals": ["happy_path", "error_path"] }
Benefits: Grounds LLM in facts, reducing hallucinations.
2. Prompt Design
Prompts are modular templates injected with CIG data, optimized via few-shot examples and chain-of-thought (CoT). Design principles:
- Specificity: Include CIG verbatim.
- Diversity: Generate N variants (e.g., 10-50) targeting coverage quadrants (happy, sad, edge, weird).
- Constraints: Enforce test framework (e.g., pytest, JUnit), assertions, mocks.
- Iteration: If verification fails, refine prompts with feedback.
Core Prompt Template (Python/pytest example):
You are a expert pytest test writer. Generate {num_tests} diverse unit tests for this function: {CIG} Requirements: - Use pytest framework. - Cover: {coverage_goals}. - Include assertions for outputs, exceptions, side-effects. - Edge cases: {edges}. - Few-shot example: def test_divide_happy(): assert divide(10, 2) == 5.0 Output ONLY valid pytest code, no explanations. Format as list of functions.Advanced Variants:
- Boundary-Focused: "Emphasize inputs like {boundaries}."
- Mutation-Based: "Mutate inputs semantically from {examples}."
- Negative Testing: "Force exceptions at {branches}."
Prompts are cached per function signature hash for efficiency.
3. Verification Module
Ensures generated tests are actionable. Multi-stage pipeline:
Stage Check Tool/Example Syntax Parses/compiles pytest --collect-onlyorjavacExecutability Runs without crashes pytest -v(sandboxed)Coverage Measures line/branch coverage pytest-cov(>70% threshold)Uniqueness Diffs against existing tests Git diff + semantic similarity (pre-deduplication) Mutation Score Kills mutants (via mutation-testing-tools) >80% kill rate Feedback Loop: Failed tests trigger re-prompting with error logs (e.g., "Fix: TypeError on line 5").
- Pass Criteria: 80% of batch must pass; survivors advance.
- Metrics Logged: Pass rate, coverage delta, mutant score.
4. Deduplication Module
Eliminates redundancy using hybrid approach:
- Syntactic: Normalized AST diff (e.g., via
astororgumtree). Threshold: Levenshtein < 0.3. - Semantic: Embed test code with LLM embeddings (e.g., OpenAI
text-embedding-3-smallor Sentence Transformers).- Cluster with HDBSCAN (density-based).
- Select centroid per cluster.
- Coverage-Based: Discard if it doesn't increase total coverage >1%.
Efficiency: Processes 100 tests in <10s on GPU/CPU.
Output: Ranked list (best-first by coverage/mutant score).
5. Developer Workflow Integration
GTCS is workflow-agnostic, with first-class support for:
-
CLI:
gtcs generate src/my_module.py --lang python --llm gpt-4o --output tests/generated/ --min-coverage 80 gtcs verify tests/generated/ --add-to-ci gtcs dedup --threshold 0.2 -
IDE Plugin (VS Code/JetBrains):
- Right-click function → "Generate Tests".
- Inline diff viewer for review/merge.
- Auto-commit hook:
gtcs ci-check.
-
CI/CD Hooks (GitHub Actions/Jenkins):
- name: Generate & Verify Tests run: gtcs generate --auto-approve-low-risk src/ && gtcs merge-existing- Fails build if coverage drops.
- PR comments with generated tests.
-
Review Loop:
- Generate → Review diffs → Accept/reject.
- Auto-label PRs: "🤖 GTCS Tests".
- Learning: Accepted tests fine-tune local LLM.
Customization:
- Config YAML: LLM provider, thresholds, languages.
- Extensibility: Plugin static analyzers/prompts.
Example Usage
Input Code (Python):
def divide(a: float, b: float) -> float: if b == 0: raise ZeroDivisionError("Cannot divide by zero") return a / bGenerated Tests (Top 3 after dedup/verification):
def test_divide_happy_path(): assert divide(10.0, 2.0) == 5.0 def test_divide_zero_denominator(): with pytest.raises(ZeroDivisionError, match="Cannot divide by zero"): divide(5.0, 0.0) def test_divide_negative(): assert divide(-10.0, 2.0) == -5.0Metrics: 100% branch coverage, 95% mutant score.
Implementation Roadmap & Tech Stack
- Core: Python (FastAPI for API mode), Rust for perf-critical dedup.
- LLMs: LangChain/LlamaIndex for orchestration.
- Static Analysis: Tree-sitter + language servers.
- MVP: 1 month (Python/JS focus).
- Metrics for Success: 2x test velocity, 30% coverage uplift (benchmarked on OSS repos).
GTCS transforms test writing from drudgery to augmentation, scaling human creativity with AI precision. Open-source prototype forthcoming from Trailblazer Labs.
Sherlock, Trailblazer Labs
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.