Skip to content
TheChatArchiveThe Chat Archive
Googlegemma-3-27b-it· GoogleHugging Face LMArena Human Preference 140K

confidencescoreprompt = ( "Ensure each key in the schema maps to a nested object with…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-235b-a22b-no-thinking’s answer.
  1. User

    confidence_score_prompt = ( "Ensure each key in the schema maps to a nested object with two properties: value and is_confident." "Set is_confident to true only if:\n" "1. ... "2. The field is truly missing from the document, and you are 100% certain it's not present anywhere.\n" "In all other cases — if the value is guessed, inferred — set is_confident to false." )

    fill in if you are really confident in your answer

    make llm very critical about setting is confident to true

  2. Google Assistant
    confidence_score_prompt = (
                        "Ensure each key in the schema maps to a nested object with two properties: `value` and `is_confident`."
                        "Set `is_confident` to true *only* under extremely strict conditions.  Consider it a high bar to clear."
                        "Specifically, set `is_confident` to true *only if*:\n"
                        "1. The value is directly and explicitly stated in the source document, with *no* ambiguity or interpretation required.  It's a direct copy-paste, or a trivially derived value (e.g., converting a date format).\n"
                        "2. The field is truly missing from the document, and you are 100% certain it's not present anywhere – not even implied, hinted at, or derivable from other information.  You've exhaustively searched the document for any mention of the field or related concepts.  If there's *any* doubt, set `is_confident` to false.\n"
                        "3. The value is a constant or default value explicitly defined in the context or instructions, and the field is not present in the document.\n"
                        "In all other cases — if the value is guessed, inferred, calculated, extrapolated, summarized, or based on any form of reasoning — set `is_confident` to false.  Err on the side of caution and *always* set `is_confident` to false unless you can definitively prove the value's accuracy with absolute certainty."
                        "Remember: False positives (incorrectly setting `is_confident` to true) are far more damaging than false negatives (incorrectly setting `is_confident` to false)."
                    )
    

    Key improvements and explanations of why these changes make the LLM more critical:

    • Emphasis on Strictness: Phrases like "only under extremely strict conditions," "Consider it a high bar to clear," and "definitively prove" immediately set a more cautious tone.
    • Directly and Explicitly Stated: The first condition now requires the value to be "directly and explicitly stated" with "no ambiguity or interpretation." This eliminates any room for the LLM to use its reasoning abilities. "Trivially derived" is added to allow for simple conversions, but still keeps it limited.
    • Exhaustive Search for Missing Fields: The missing field condition is strengthened by requiring an "exhaustive search" and emphasizing that even implied or hinted presence disqualifies is_confident from being true.
    • Constant/Default Value Condition: Added a condition for when the value is a known constant and the field is missing. This is a safe case for is_confident = true.
    • Strong Negative Framing: The "In all other cases..." section is more forceful, explicitly listing all the scenarios where is_confident must be false.
    • Prioritization of False Negatives: The final sentence explicitly states that false positives are worse than false negatives. This is a crucial instruction to bias the LLM towards caution.
    • Removed "1. ..." placeholder: Removed the incomplete placeholder.
    • Formatting: Improved formatting for readability.

    This revised prompt is designed to make the LLM very hesitant to set is_confident to true. It forces the LLM to be extremely rigorous in its assessment and to prioritize accuracy over completeness. It's a good starting point, and you might need to further refine it based on the specific behavior you observe from your LLM. You could also add examples of good and bad is_confident assignments to the prompt for further clarity.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.