Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek-v3-0324· DeepSeekHugging Face LMArena Human Preference 140K

do any llm's have the ability to check their own code for errors?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-235b-a22b-no-thinking’s answer.
  1. User

    do any llm's have the ability to check their own code for errors?

  2. DeepSeek Assistant

    Yes! Some LLMs (like Claude, GPT-4, and Gemini) can self-check their own code for errors, but with important limitations. Here’s how they work and what to watch out for:


    🔍 How LLMs Self-Check Code

    1. Syntax & Basic Logic Errors

      • They can detect missing brackets, undefined variables, or obvious logical flaws.
      • Example:
        def add(a, b):
            return a + b
        print(add(5))  # ❌ Missing argument
        
        An LLM might catch: "add() requires 2 arguments, but only 1 was given."
    2. Static Analysis (Sometimes)

      • Some models simulate a linter (like ESLint, Pylint) to spot bad practices.
      • Example:
        let x = 10;
        if (x = 5) { ... }  // ❌ Assignment in condition
        
        An LLM might warn: "Did you mean == instead of =?"
    3. Runtime Simulation (Limited)

      • Advanced models (Claude 3 Opus, GPT-4 Turbo) can "imagine" code execution step-by-step.
      • Example:
        for i in range(5):
            print(i)
            i += 2  # ❌ Useless in Python loops
        
        An LLM might note: "i += 2 has no effect because range(5) controls iteration."
    4. Self-Critique Prompts

      • You can explicitly ask:
        "Review this code for bugs and suggest fixes."
        "Is there a memory leak in this C++ snippet?"

    ⚠️ Limitations

    • No True Execution – LLMs don’t actually run code; they predict based on patterns.
    • False Positives/Negatives – They might miss subtle bugs or hallucinate non-issues.
    • Complexity Threshold – Deep algorithmic flaws (e.g., race conditions, cryptographic errors) often slip through.

    🛠️ Best Practices for Self-Checking

    1. Ask Explicitly

      • "What’s wrong with this code?"
      • "How would you improve this function?"
    2. Compare Multiple LLMs

      • Run the same code through Claude, GPT-4, and Gemini to cross-check results.
    3. Use Tools Alongside LLMs

      • Pair LLM feedback with:
        • Linters (ESLint, Pylint)
        • Static Analyzers (SonarQube, CodeQL)
        • Unit Tests (Jest, pytest)
    4. Iterative Debugging

      • "I got this error: [paste error]. How do I fix it?"

    📌 Example: Claude Checking Its Own Raymarcher Code

    If I review my earlier DDA raymarcher, I might ask myself:

    • "Does the rayMarch function handle edge cases?"
      → "No max depth check could cause infinite loops."
    • "Is the normal calculation numerically stable?"
      → "Using a smaller h (epsilon) might improve precision."

    🔥 Top LLMs for Code Review

    ModelStrengthsWeaknesses
    Claude 3Strong logic analysis, fewer hallucinationsLess familiar with niche libs
    GPT-4 TurboBroad language support, good at explainingCan be verbose
    Gemini 1.5Handles long context wellMisses subtle C++ bugs

    For critical code, always validate with real execution (e.g., CI/CD pipelines). LLMs are assistants, not replacements for testing.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.