Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· OpenAIHugging Face Arena Human Preference 55K

Here are some benchmarks I was thinking about to assess large language models, can you…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-3.5-turbo-1106’s answer.
  1. User

    Here are some benchmarks I was thinking about to assess large language models, can you please come up with some specific prompts I could use: '''Category Sub-Category How to Measure Instruction Following Complexity Handling Create prompts with multi-part instructions and assess if all parts are addressed correctly. Adaptive Execution Provide feedback or corrections to initial responses and evaluate how the model adjusts. Contextual Understanding Use instructions with implicit requirements and see if the model infers them correctly. Accuracy and Precision Check the precision of the model's responses against the specific instructions given. Reasoning Logical Analysis Test with puzzles or problems that require logical deduction. Abstract Thinking Pose abstract, theoretical, or hypothetical scenarios for the model to interpret or explain. Coding Application Conceptualization Evaluate the model's ability to outline a functional application concept from a given idea. User Experience Design Assess the model's suggestions for user interface and user interaction designs. Application Development Monitor the model's ability to develop an application iteratively, enhancing features progressively. Creativity Idea Generation Prompt the model to generate unique ideas in various domains (e.g., business, art). Divergent Thinking Challenge the model with scenarios that require unconventional solutions. Multi-Step Conversation Context Retention Engage in extended conversations and see how well the model recalls earlier information. Coherence Over Time Assess the consistency and relevance of the model's responses throughout an extended dialogue.'''

  2. Assistant

    Sure! Here are specific prompts you could use for each of the sub-categories you mentioned:

    Instruction Following - Complexity Handling

    • "Please provide a three-course meal plan for a vegan dinner party, including an appetizer, main course, and dessert, each with a list of ingredients and a brief description of the preparation steps. Also, suggest a theme for the decoration and music playlist that complements the vibe of the evening."

    Adaptive Execution

    • Initial prompt: "Write a short story about a detective solving a mystery in space." Follow-up: "Actually, make the detective an AI and include a plot twist involving a misunderstood alien culture."

    Contextual Understanding

    • "Explain the steps a person should take to secure their home before leaving on a two-week vacation, considering they own smart home devices and have a pet."

    Accuracy and Precision

    • "Calculate the amount of paint needed to cover a room that is 12 feet long, 10 feet wide, and 9 feet high, with two coats of paint, assuming one gallon covers 400 square feet."

    Reasoning - Logical Analysis

    • "If all roses are flowers and some flowers fade quickly, can we conclude that some roses fade quickly? Explain your reasoning."

    Abstract Thinking

    • "Discuss the implications of teleportation technology on global economies and interpersonal relationships."

    Coding - Application Conceptualization

    • "Conceptualize an application that helps users learn a new language by matching them with native speakers for practice sessions. Outline the core features and technologies needed."

    User Experience Design

    • "Design a user interface for a mobile app that helps people with visual impairments navigate public transportation. Describe the key features that would enhance their experience."

    Application Development

    • "Imagine you're developing a fitness tracking app. Describe the process of adding a feature that allows users to set and track their weekly exercise goals."

    Creativity - Idea Generation

    • "Generate five unique business ideas that leverage virtual reality technology to enhance user experience in education."

    Divergent Thinking

    • "Come up with a solution to reduce city traffic congestion without constructing new roads or increasing public transportation."

    Multi-Step Conversation - Context Retention

    • Engage in a conversation about a fictional character's backstory, motivations, and future actions, asking for more details and recalling earlier information over several exchanges.

    Coherence Over Time

    • Initiate a dialogue that spans multiple topics, including a hobby, a recent news event, and planning a hypothetical event, while ensuring the model stays on topic and maintains coherence throughout the conversation.

    By using these prompts, you can assess the capabilities of a large language model in various dimensions, from handling complex tasks to maintaining coherence over extended interactions.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.