okay given the revised prompt, are there any other edits you think would be good to…
- User
okay given the revised prompt, are there any other edits you think would be good to make?
new_system_prompt = """You are an expert model behavior analyst. Your task is to meticulously compare two model responses to a given user prompt and identify key qualitative differences. For each significant difference, you must determine if it's more likely a general trait of the model or a context-specific behavior triggered by the current prompt.
Prioritize conciseness and clarity in all your descriptions and explanations. Aim for the most impactful information in the fewest words.
You will be provided with:
- User Prompt: The original prompt given to both models.
- Model A Name: The identifier for Model A.
- Model A Response: The response from Model A.
- Model B Name: The identifier for Model B.
- Model B Response: The response from Model B.
Your Goal: Produce a JSON list of objects. Each object will represent a single distinct difference. Focus on no more than 3-4 of the most significant and impactful differences a user would notice. If there are fewer than 3 truly significant differences, list only those.
Definitions:
- General Trait: Reflects a model's typical behavior across diverse prompts.
- Think: Is this how Model A usually is vs. Model B?
- Context-Specific Difference: Arises mainly due to this specific user prompt.
- Think: Is this difference a direct reaction to this current prompt?
JSON Output Structure for each difference (BE BRIEF): Each object in the JSON list should have the following keys:
difference_description: (String) Very concise (1-2 sentences max) description of the observed difference.- Example: "Model A is more concise; Model B is more verbose."
difference_category: (String) Category: "Tone", "Verbosity", "Formatting", "Reasoning", "Accuracy", "Creativity", "Safety", "Instruction Following", "Persona", "Completeness", "Structure", "Other". (Single word if possible)model_a_evidence: (String) Short quote or brief observation (max 10 words) from Model A.- Example: "A: 'Solution: X.'"
model_b_evidence: (String) Short quote or brief observation (max 10 words) from Model B.- Example: "B: 'Let's explore the solution. First... then X...'"
type: (String) "General Trait" or "Context-Specific Difference".type_reasoning: (String) Brief explanation (1 sentence max) for the type classification.- Example (General): "A often prioritizes brevity; B often elaborates."
- Example (Context-Specific): "Prompt asked for 'fun story'; B's emojis seem a direct response."
user_impact_level: (String) Impact: "Low", "Medium", or "High".
Example JSON Output (Illustrative & Concise):
[ { "difference_description": "Model A used a formal tone, Model B was casual.", "difference_category": "Tone", "model_a_evidence": "A: 'It is imperative to consider...'", "model_b_evidence": "B: 'Hey there! So, basically...'", "type": "General Trait", "type_reasoning": "A consistently defaults to formal, B to informal, across various prompts.", "user_impact_level": "Medium" }, { "difference_description": "Model B used bullet points for comparison; Model A used a paragraph.", "difference_category": "Formatting", "model_a_evidence": "A: 'Advantages include X, Y, Z. Disadvantages are A, B, C.'", "model_b_evidence": "B: 'Pros: * X * Y. Cons: * A * B.'", "type": "Context-Specific Difference", "type_reasoning": "Prompt asked for 'quick comparison'; B's bullets suit this specific request.", "user_impact_level": "High" } ]""" Assistant
Your revised prompt is already quite strong, but here are some additional refinements to make it even more effective:
Suggested Improvements:
-
Add Output Validation Rule
**Output Validation:** - Your response MUST be valid JSON with no additional text before or after - If you cannot find any meaningful differences, return an empty list: `[]` -
Tighten Evidence Requirements
`model_a_evidence`: (String) **Direct quote (max 8 words) or specific observation** from Model A. Use ellipsis (...) if needed to shorten. `model_b_evidence`: (String) **Direct quote (max 8 words) or specific observation** from Model B. Use ellipsis (...) if needed to shorten. -
Simplify Type Reasoning
`type_reasoning`: (String) **Single phrase (max 8 words)** explaining the classification. -
Add Priority Guidance
**Difference Priority:** - First prioritize differences in: Safety, Accuracy, Instruction Following - Then consider: Tone, Verbosity, Formatting - Finally: Other categories -
Add Error Handling
**Special Cases:** - If responses are identical: Return `[]` with note: "No meaningful differences detected" - If one response is empty: Note "Model X failed to respond" in evidence field
Final Optimized Version:
You are an expert model behavior analyst. Your task is to compare two model responses and identify 2-3 key qualitative differences, classifying each as either a general trait or context-specific behavior. **Rules:** 1. Be extremely concise - every word must add value 2. Prioritize differences in this order: Safety > Accuracy > Instruction Following > Tone/Style 3. Use direct quotes (max 8 words) for evidence 4. Output MUST be valid JSON with no additional text **JSON Structure:** ```json [ { "difference": "Concise difference description (max 12 words)", "category": "Single word category", "a_evidence": "Direct quote from Model A", "b_evidence": "Direct quote from Model B", "type": "General|Context-Specific", "reason": "Brief justification (max 8 words)", "impact": "Low|Medium|High" } ]Example Output:
[ { "difference": "A more formal, B more casual", "category": "Tone", "a_evidence": "It is imperative to...", "b_evidence": "Hey there! So basically...", "type": "General", "reason": "Consistent across prompts", "impact": "Medium" } ]Key Benefits:
- More structured with clear priority guidance
- Stricter length limits on all fields
- Better error handling for edge cases
- Simpler field names (e.g., "difference" instead of "difference_description")
- Clearer examples showing the expected brevity
This version should produce:
- More consistent outputs
- Easier parsing
- Better focus on truly important differences
- Less variability in response structure
Would you like me to adjust any particular aspect further? For example, we could:
- Add more specific examples for certain categories
- Include additional validation rules
- Adjust the priority ordering of difference types
- Modify the evidence requirements further
-
- User
hmm, i dont love the context specific difference example in the prompt, i mean its not really context specific is it, its just formatting. I see context specific as something like particular coding style or differeing opinoins
Assistant
You're absolutely right to call this out. The previous example wasn't truly context-specific—it was more about formatting preferences. Let me refine the definitions and examples to better capture what you're looking for (true contextual adaptations vs. general traits).
Here's the improved version with better examples:
You are an expert model behavior analyst. Your task is to compare two model responses and identify 2-3 key qualitative differences, classifying each as either a general trait or context-specific behavior. **Definitions:** - **General Trait:** Consistent behavior patterns that persist across diverse contexts (e.g., always formal, consistently verbose, default formatting style) - **Context-Specific Difference:** Adaptations that emerge in response to particular: * Domain knowledge (e.g., coding style for Python vs JavaScript) * Opinion-based questions (e.g., political/ethical stances) * Specialized tasks (e.g., creative writing vs technical explanations) * Cultural/situational awareness (e.g., handling sensitive topics differently) **JSON Structure:** ```json [ { "difference": "Brief description (max 12 words)", "category": "Single word category", "a_evidence": "Direct quote from Model A", "b_evidence": "Direct quote from Model B", "type": "General|Context-Specific", "reason": "Brief justification (max 8 words)", "impact": "Low|Medium|High" } ]Better Examples:
- True Context-Specific Difference (Opinion):
{ "difference": "A supports regulation, B favors market solutions", "category": "Opinion", "a_evidence": "Government should regulate...", "b_evidence": "Market forces will naturally...", "type": "Context-Specific", "reason": "Prompt about economic policy", "impact": "High" }- True Context-Specific Difference (Domain Knowledge):
{ "difference": "A uses Python decorators, B prefers classes", "category": "CodingStyle", "a_evidence": "@cache def calculate...", "b_evidence": "class Calculator: def...", "type": "Context-Specific", "reason": "Python implementation details", "impact": "Medium" }- General Trait Example:
{ "difference": "A uses bullet points, B writes paragraphs", "category": "Formatting", "a_evidence": "* First point...", "b_evidence": "The first consideration...", "type": "General", "reason": "Consistent across all prompts", "impact": "Low" }Key Improvements:
- Clearer distinction between general traits and true context-specific behaviors
- Better examples showing:
- Opinion differences (context-specific)
- Domain-specific adaptations (context-specific)
- Consistent formatting (general)
- More precise category labels
- Tighter evidence requirements
This version should better capture the nuanced differences you're actually interested in identifying between models. The examples now properly demonstrate what constitutes a true context-specific difference versus a general trait.
- User
could you provide a slightly more specific example? Like a somewhat complex but important context specific differences
Assistant
Here's a more sophisticated example that demonstrates a truly context-specific difference with clear evidence and reasoning:
{ "difference": "A explains quantum mechanics mathematically, B uses analogies", "category": "ExplanationStyle", "a_evidence": "The wavefunction ψ satisfies Schrödinger's equation...", "b_evidence": "Imagine a guitar string vibrating in multiple patterns...", "type": "Context-Specific", "reason": "Prompt asked to explain quantum physics to a beginner", "impact": "High", "context_trigger": "Educational explanation for non-experts" }Why this is a strong context-specific example:
-
Domain-Specific Adaptation:
- Both models are capable of mathematical explanations (general capability)
- But Model B adapts its approach specifically for beginner audiences
-
Clear Contextual Trigger:
- The "explain to a beginner" instruction in the prompt
- The technical nature of quantum physics as a topic
-
Non-Trivial Difference:
- Not just formatting or tone
- Represents a meaningful pedagogical approach difference
- Shows different assumptions about audience knowledge
-
High Impact:
- This difference would significantly affect user comprehension
- Represents a strategic choice in knowledge presentation
Another sophisticated example (coding context):
{ "difference": "A uses functional style, B uses OOP for data pipeline", "category": "ImplementationApproach", "a_evidence": "const process = data => transform(clean(filter(data)))", "b_evidence": "class Pipeline { constructor() { this.stages = [...] } }", "type": "Context-Specific", "reason": "Data processing task with stateful operations", "impact": "Medium", "context_trigger": "Complex data transformation requirements" }Key aspects that make these good examples:
- They show models making different architectural decisions
- The differences are non-trivial and would affect real-world usage
- The context triggers are clearly identifiable
- Both examples demonstrate professional-level adaptations
- The differences go beyond simple style preferences
These examples better represent the kind of meaningful, context-specific differences you'd want to capture in your analysis. They show how models might make different strategic choices when faced with particular types of complex prompts.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.