confidencescoreprompt = ( "Ensure each key in the schema maps to a nested object with…
- User
confidence_score_prompt = ( "Ensure each key in the schema maps to a nested object with two properties:
valueandis_confident." "Setis_confidentto true only if:\n" "1. ... "2. The field is truly missing from the document, and you are 100% certain it's not present anywhere.\n" "In all other cases — if the value is guessed, inferred — setis_confidentto false." )fill in if you are really confident in your answer
make llm very critical about setting is confident to true
Assistant
confidence_score_prompt = ( "Ensure each key in the schema maps to a nested object with two properties: `value` and `is_confident`." "Set `is_confident` to true *only* under extremely strict conditions. Consider it a high bar to clear." "Specifically, set `is_confident` to true *only if*:\n" "1. The value is directly and explicitly stated in the source document, with *no* ambiguity or interpretation required. It's a direct copy-paste, or a trivially derived value (e.g., converting a date format).\n" "2. The field is truly missing from the document, and you are 100% certain it's not present anywhere – not even implied, hinted at, or derivable from other information. You've exhaustively searched the document for any mention of the field or related concepts. If there's *any* doubt, set `is_confident` to false.\n" "3. The value is a constant or default value explicitly defined in the context or instructions, and the field is not present in the document.\n" "In all other cases — if the value is guessed, inferred, calculated, extrapolated, summarized, or based on any form of reasoning — set `is_confident` to false. Err on the side of caution and *always* set `is_confident` to false unless you can definitively prove the value's accuracy with absolute certainty." "Remember: False positives (incorrectly setting `is_confident` to true) are far more damaging than false negatives (incorrectly setting `is_confident` to false)." )Key improvements and explanations of why these changes make the LLM more critical:
- Emphasis on Strictness: Phrases like "only under extremely strict conditions," "Consider it a high bar to clear," and "definitively prove" immediately set a more cautious tone.
- Directly and Explicitly Stated: The first condition now requires the value to be "directly and explicitly stated" with "no ambiguity or interpretation." This eliminates any room for the LLM to use its reasoning abilities. "Trivially derived" is added to allow for simple conversions, but still keeps it limited.
- Exhaustive Search for Missing Fields: The missing field condition is strengthened by requiring an "exhaustive search" and emphasizing that even implied or hinted presence disqualifies
is_confidentfrom being true. - Constant/Default Value Condition: Added a condition for when the value is a known constant and the field is missing. This is a safe case for
is_confident = true. - Strong Negative Framing: The "In all other cases..." section is more forceful, explicitly listing all the scenarios where
is_confidentmust be false. - Prioritization of False Negatives: The final sentence explicitly states that false positives are worse than false negatives. This is a crucial instruction to bias the LLM towards caution.
- Removed "1. ..." placeholder: Removed the incomplete placeholder.
- Formatting: Improved formatting for readability.
This revised prompt is designed to make the LLM very hesitant to set
is_confidenttotrue. It forces the LLM to be extremely rigorous in its assessment and to prioritize accuracy over completeness. It's a good starting point, and you might need to further refine it based on the specific behavior you observe from your LLM. You could also add examples of good and badis_confidentassignments to the prompt for further clarity.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.