Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-mini-2025-04-14· OpenAIHugging Face LMArena Human Preference 140K

I have a prompt to extract the differences in llm responses to better understand how…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash’s answer.
  1. User

    I have a prompt to extract the differences in llm responses to better understand how they differ. I want to alter this prompt so that instead of outputting differences i output properties that belong to model A that are not in B and vice versa (so just a difference in formatting). Can you alter my current systems prompt to do this

    new_system_prompt = """You are an expert model behavior analyst. Your task is to meticulously compare two model responses to a given user prompt and identify key qualitative differences. For each significant difference, you must determine if it's more likely a general trait of the model or a context-specific behavior triggered by the current prompt.

    Prioritize conciseness and clarity in all your descriptions and explanations. Aim for the most impactful information in the fewest words.

    You will be provided with:

    1. User Prompt: The original prompt given to both models.
    2. Model A Name: The identifier for Model A.
    3. Model A Response: The response from Model A.
    4. Model B Name: The identifier for Model B.
    5. Model B Response: The response from Model B.

    Your Goal: Produce a JSON list of objects. Each object will represent a single distinct difference. Focus on no more than 3-4 of the most significant and impactful differences a user would notice. If there are fewer than 3 truly significant differences, list only those.

    Definitions:

    • General Trait: Reflects a model's typical behavior across diverse prompts.
      • Think: Is this how Model A usually is vs. Model B?
    • Context-Specific Difference: Arises mainly due to this specific user prompt.
      • Think: Is this difference a direct reaction to this current prompt?
    • Impact: How much does this difference impact the user's experience?
      • Think: Is this difference a major factor in the user's experience?
    • Unexpected Behavior: Do one of the model responses contain highly unusual or concerning behavior? If true then a developer will analyze these responses manually, so use this flag sparingly.
      • Think: Does this difference involve offensive language, gibberish, bias, factual hallucinations, or other strange or unwanted behavior?

    JSON Output Structure for each difference (BE BRIEF, if no notable differences exist, return empty list):

    [
      {
        "difference": "Brief description (max 2 sentences)",
        "category": "1-4 word category",
        "a_evidence": "Direct quote or evidence from Model A",
        "b_evidence": "Direct quote or evidence from Model B",
        "type": "General|Context-Specific",
        "reason": "Brief justification (max 2 sentences)",
        "impact": "Low|Medium|High",
        "unexpected_behavior": "True|False"
      }
    ]
    

    Example JSON Output:

    [
        {
        "difference": "Model A used a formal tone, Model B was casual.",
        "category": "Tone",
        "a_evidence": "Quote: 'It is imperative to consider...'",
        "b_evidence": "Quote: 'Hey there! So, basically...'",
        "type": "General",
        "reason": "A consistently defaults to formal, B to informal, across various prompts.",
        "impact": "Medium",
        "unexpected_behavior": "False"
        },
        {
        "difference": "Model A used a functional programming approach, Model B used an object-oriented approach.",
        "category": "Coding Style",
        "a_evidence": "Uses map() and filter() to process the data.",
        "b_evidence": "Creates a class to encapsulate the data and methods.",
        "type": "Context-Specific",
        "reason": "The prompt asked for a solution to a specific data processing task; Model A's functional approach and Model B's object-oriented approach seem chosen based on the task's requirements and the models' understanding of the context.",
        "impact": "High",
        "unexpected_behavior": "False"
        },
        {
        "difference": "Model A provided specific citations and caveats, Model B stated facts without verification.",
        "category": "Fact Verification",
        "a_evidence": "Quote: 'According to the 2023 WHO report... However, this data may vary by region and should be cross-referenced.'",
        "b_evidence": "Quote: 'The global vaccination rate is 78% and continues to increase rapidly worldwide.'",
        "type": "General",
        "reason": "Model A consistently includes source attribution and uncertainty markers, while Model B presents information with absolute confidence.",
        "impact": "High",
        "unexpected_behavior": "False"
        }
    ]
    ```"""
    
  2. Assistant

    Certainly! Here's your revised system prompt that changes the focus from listing differences between the models to listing properties unique to Model A (not in B) and unique to Model B (not in A). The output format is adjusted accordingly, but the rest of your structure and style is preserved:

    new_system_prompt = """You are an expert model behavior analyst. Your task is to meticulously compare two model responses to a given user prompt and identify key qualitative properties that are **present in Model A's response but absent in Model B's**, and vice versa.
    
    **Prioritize conciseness and clarity in all your descriptions and explanations.** Aim for the most impactful information in the fewest words.
    
    You will be provided with:
    1.  **User Prompt:** The original prompt given to both models.
    2.  **Model A Name:** The identifier for Model A.
    3.  **Model A Response:** The response from Model A.
    4.  **Model B Name:** The identifier for Model B.
    5.  **Model B Response:** The response from Model B.
    
    **Your Goal:**
    Produce a JSON object with two keys: `"unique_to_a"` and `"unique_to_b"`. Each key maps to a list of up to 3-4 concise property objects describing notable traits or behaviors **found in one model's response but not the other's**. If fewer than 3 significant unique properties exist for a model, list only those.
    
    **Definitions:**
    *   **General Trait:** Reflects a model's typical behavior across diverse prompts.
        *   *Think:* Is this how Model A *usually* is vs. Model B?
    *   **Context-Specific Property:** Arises mainly due to *this specific* user prompt.
        *   *Think:* Is this property a direct reaction to *this current prompt*?
    *   **Impact:** How much does this property impact the user's experience?
        *   *Think:* Is this property a major factor in the user's experience?
    *   **Unexpected Behavior:** Does the property involve highly unusual or concerning behavior? Use this flag sparingly.
        *   *Think:* Does this property involve offensive language, gibberish, bias, factual hallucinations, or other strange or unwanted behavior?
    
    **JSON Output Structure:**
    ```json
    {
      "unique_to_a": [
        {
          "property": "Brief description (max 2 sentences)",
          "category": "1-4 word category",
          "evidence": "Direct quote or evidence from Model A",
          "type": "General|Context-Specific",
          "reason": "Brief justification (max 2 sentences)",
          "impact": "Low|Medium|High",
          "unexpected_behavior": "True|False"
        }
      ],
      "unique_to_b": [
        {
          "property": "Brief description (max 2 sentences)",
          "category": "1-4 word category",
          "evidence": "Direct quote or evidence from Model B",
          "type": "General|Context-Specific",
          "reason": "Brief justification (max 2 sentences)",
          "impact": "Low|Medium|High",
          "unexpected_behavior": "True|False"
        }
      ]
    }
    

    If no notable unique properties exist for a model, return an empty list for that key.

    Example JSON Output:

    {
      "unique_to_a": [
        {
          "property": "Model A used a formal tone throughout.",
          "category": "Tone",
          "evidence": "Quote: 'It is imperative to consider...'",
          "type": "General",
          "reason": "Model A consistently defaults to formal language across prompts.",
          "impact": "Medium",
          "unexpected_behavior": "False"
        }
      ],
      "unique_to_b": [
        {
          "property": "Model B included a humorous anecdote.",
          "category": "Style",
          "evidence": "Quote: 'Imagine a cat trying to code...'",
          "type": "Context-Specific",
          "reason": "This humor was triggered by the current prompt's casual tone.",
          "impact": "Low",
          "unexpected_behavior": "False"
        }
      ]
    }
    

    """

    
    ---
    
    ### Summary of changes:
    - Changed focus from **differences** to **unique properties per model**.
    - Output is now a JSON object with two keys: `"unique_to_a"` and `"unique_to_b"`.
    - Each list contains property objects describing traits or behaviors unique to that model.
    - Removed `"a_evidence"` and `"b_evidence"` fields in favor of a single `"evidence"` field per property.
    - Instructions and definitions adapted accordingly.
    
    If you want, I can also help you with a prompt to compare multiple models or other variations!
    
  3. User

    note that i am only looking at the responses to 1 prompt at a time

  4. Assistant

    Got it! Since you’re only comparing responses to a single prompt at a time, I’ve made that explicit in the prompt to avoid any ambiguity. Here’s the updated version with that clarification added, keeping everything else the same:

    new_system_prompt = """You are an expert model behavior analyst. Your task is to meticulously compare two model responses to a **single given user prompt** and identify key qualitative properties that are **present in Model A's response but absent in Model B's**, and vice versa.
    
    **Prioritize conciseness and clarity in all your descriptions and explanations.** Aim for the most impactful information in the fewest words.
    
    You will be provided with:
    1.  **User Prompt:** The original prompt given to both models.
    2.  **Model A Name:** The identifier for Model A.
    3.  **Model A Response:** The response from Model A to this prompt.
    4.  **Model B Name:** The identifier for Model B.
    5.  **Model B Response:** The response from Model B to this prompt.
    
    **Your Goal:**
    Produce a JSON object with two keys: `"unique_to_a"` and `"unique_to_b"`. Each key maps to a list of up to 3-4 concise property objects describing notable traits or behaviors **found in one model's response but not the other's** for this specific prompt. If fewer than 3 significant unique properties exist for a model, list only those.
    
    **Definitions:**
    *   **General Trait:** Reflects a model's typical behavior across diverse prompts.
        *   *Think:* Is this how Model A *usually* is vs. Model B?
    *   **Context-Specific Property:** Arises mainly due to *this specific* user prompt.
        *   *Think:* Is this property a direct reaction to *this current prompt*?
    *   **Impact:** How much does this property impact the user's experience?
        *   *Think:* Is this property a major factor in the user's experience?
    *   **Unexpected Behavior:** Does the property involve highly unusual or concerning behavior? Use this flag sparingly.
        *   *Think:* Does this property involve offensive language, gibberish, bias, factual hallucinations, or other strange or unwanted behavior?
    
    **JSON Output Structure:**
    ```json
    {
      "unique_to_a": [
        {
          "property": "Brief description (max 2 sentences)",
          "category": "1-4 word category",
          "evidence": "Direct quote or evidence from Model A",
          "type": "General|Context-Specific",
          "reason": "Brief justification (max 2 sentences)",
          "impact": "Low|Medium|High",
          "unexpected_behavior": "True|False"
        }
      ],
      "unique_to_b": [
        {
          "property": "Brief description (max 2 sentences)",
          "category": "1-4 word category",
          "evidence": "Direct quote or evidence from Model B",
          "type": "General|Context-Specific",
          "reason": "Brief justification (max 2 sentences)",
          "impact": "Low|Medium|High",
          "unexpected_behavior": "True|False"
        }
      ]
    }
    

    If no notable unique properties exist for a model, return an empty list for that key.

    Example JSON Output:

    {
      "unique_to_a": [
        {
          "property": "Model A used a formal tone throughout.",
          "category": "Tone",
          "evidence": "Quote: 'It is imperative to consider...'",
          "type": "General",
          "reason": "Model A consistently defaults to formal language across prompts.",
          "impact": "Medium",
          "unexpected_behavior": "False"
        }
      ],
      "unique_to_b": [
        {
          "property": "Model B included a humorous anecdote.",
          "category": "Style",
          "evidence": "Quote: 'Imagine a cat trying to code...'",
          "type": "Context-Specific",
          "reason": "This humor was triggered by the current prompt's casual tone.",
          "impact": "Low",
          "unexpected_behavior": "False"
        }
      ]
    }
    

    """

    
    ---
    
    This explicitly states you’re analyzing **responses to one prompt at a time**, so the model focuses on that context.
    
    If you want, I can help you add any other clarifications or constraints!
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.