do any llm's have the ability to check their own code for errors?
- User
do any llm's have the ability to check their own code for errors?
Assistant
Yes! Some LLMs (like Claude, GPT-4, and Gemini) can self-check their own code for errors, but with important limitations. Here’s how they work and what to watch out for:
🔍 How LLMs Self-Check Code
-
Syntax & Basic Logic Errors
- They can detect missing brackets, undefined variables, or obvious logical flaws.
- Example:
An LLM might catch: "def add(a, b): return a + b print(add(5)) # ❌ Missing argumentadd()requires 2 arguments, but only 1 was given."
-
Static Analysis (Sometimes)
- Some models simulate a linter (like ESLint, Pylint) to spot bad practices.
- Example:
An LLM might warn: "Did you meanlet x = 10; if (x = 5) { ... } // ❌ Assignment in condition==instead of=?"
-
Runtime Simulation (Limited)
- Advanced models (Claude 3 Opus, GPT-4 Turbo) can "imagine" code execution step-by-step.
- Example:
An LLM might note: "for i in range(5): print(i) i += 2 # ❌ Useless in Python loopsi += 2has no effect becauserange(5)controls iteration."
-
Self-Critique Prompts
- You can explicitly ask:
"Review this code for bugs and suggest fixes."
"Is there a memory leak in this C++ snippet?"
- You can explicitly ask:
⚠️ Limitations
- No True Execution – LLMs don’t actually run code; they predict based on patterns.
- False Positives/Negatives – They might miss subtle bugs or hallucinate non-issues.
- Complexity Threshold – Deep algorithmic flaws (e.g., race conditions, cryptographic errors) often slip through.
🛠️ Best Practices for Self-Checking
-
Ask Explicitly
- "What’s wrong with this code?"
- "How would you improve this function?"
-
Compare Multiple LLMs
- Run the same code through Claude, GPT-4, and Gemini to cross-check results.
-
Use Tools Alongside LLMs
- Pair LLM feedback with:
- Linters (ESLint, Pylint)
- Static Analyzers (SonarQube, CodeQL)
- Unit Tests (Jest, pytest)
- Pair LLM feedback with:
-
Iterative Debugging
- "I got this error: [paste error]. How do I fix it?"
📌 Example: Claude Checking Its Own Raymarcher Code
If I review my earlier DDA raymarcher, I might ask myself:
- "Does the
rayMarchfunction handle edge cases?"
→ "No max depth check could cause infinite loops." - "Is the normal calculation numerically stable?"
→ "Using a smallerh(epsilon) might improve precision."
🔥 Top LLMs for Code Review
Model Strengths Weaknesses Claude 3 Strong logic analysis, fewer hallucinations Less familiar with niche libs GPT-4 Turbo Broad language support, good at explaining Can be verbose Gemini 1.5 Handles long context well Misses subtle C++ bugs For critical code, always validate with real execution (e.g., CI/CD pipelines). LLMs are assistants, not replacements for testing.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.