Here are some benchmarks I was thinking about to assess large language models, can you…
- User
Here are some benchmarks I was thinking about to assess large language models, can you please come up with some specific prompts I could use: '''Category Sub-Category How to Measure Instruction Following Complexity Handling Create prompts with multi-part instructions and assess if all parts are addressed correctly. Adaptive Execution Provide feedback or corrections to initial responses and evaluate how the model adjusts. Contextual Understanding Use instructions with implicit requirements and see if the model infers them correctly. Accuracy and Precision Check the precision of the model's responses against the specific instructions given. Reasoning Logical Analysis Test with puzzles or problems that require logical deduction. Abstract Thinking Pose abstract, theoretical, or hypothetical scenarios for the model to interpret or explain. Coding Application Conceptualization Evaluate the model's ability to outline a functional application concept from a given idea. User Experience Design Assess the model's suggestions for user interface and user interaction designs. Application Development Monitor the model's ability to develop an application iteratively, enhancing features progressively. Creativity Idea Generation Prompt the model to generate unique ideas in various domains (e.g., business, art). Divergent Thinking Challenge the model with scenarios that require unconventional solutions. Multi-Step Conversation Context Retention Engage in extended conversations and see how well the model recalls earlier information. Coherence Over Time Assess the consistency and relevance of the model's responses throughout an extended dialogue.'''
Assistant
Sure, here are some specific prompts you could use for each of the sub-categories of the benchmarks you provided:
Instruction Following:
- "Please provide step-by-step instructions on how to make a peanut butter and jelly sandwich, including any necessary ingredients and equipment."
- "Can you give me a list of all the tools and materials I would need to build a simple birdhouse, along with any specific instructions on how to use them?"
Adaptive Execution:
- "I'd like to build a simple website. Please provide me with a basic template and allow me to fill in the content. As I fill in the content, can you provide feedback on any mistakes I make or suggest improvements?"
- "I'm trying to plan a trip to a new city. Can you provide me with some initial recommendations for things to do and see, and then as I narrow down my options, can you provide more tailored suggestions based on my preferences?"
Contextual Understanding:
- "Please provide me with instructions on how to assemble a bookshelf, but only using implicit information from the context of the room where the bookshelf will be placed."
- "I'd like to plan a dinner party. Can you provide me with a list of potential dishes I could serve, along with any dietary restrictions or preferences that I should consider?"
Accuracy and Precision:
- "Please provide me with the exact measurements for cutting a piece of wood to fit a specific frame, based on the dimensions provided."
- "I'd like to write a letter to a friend. Can you provide me with a list of specific vocabulary and grammar structures that I should use to ensure my letter is clear and concise?"
Reasoning:
- "Please provide me with a list of logical deductions I can make based on the information provided about a given scenario."
- "I'd like to solve a simple puzzle. Can you provide me with a set of clues and allow me to reason my way to the solution?"
Abstract Thinking:
- "Please provide me with a hypothetical scenario and ask me to describe a potential solution or outcome."
- "I'd like to explore the concept of 'freedom' in a philosophical context. Can you provide me with a set of abstract ideas and allow me to develop my own reasoning and conclusions?"
Coding:
- "Please provide me with a set of programming concepts and ask me to outline a functional application concept based on those concepts."
- "I'd like to build a simple mobile app. Can you provide me with a list of features I could include, along with any technical requirements or considerations I should keep in mind?"
User Experience Design:
- "Please provide me with a set of design principles and ask me to create a user interface and user interaction design for a simple web application."
- "I'd like to create a user-friendly interface for a new product. Can you provide me with a list of potential features and allow me to suggest specific design elements and interactions?"
Application Development:
- "Please provide me with a set of requirements for a simple web application and ask me to develop the application iteratively, enhancing features progressively."
- "I'd like to build a simple machine learning model. Can you provide me with a set of data and allow me to develop and train the model iteratively?"
Creativity:
- "Please provide me with a set of random words or phrases and ask me to generate a unique idea or concept based on those prompts."
- "I'd like to generate a list of potential business ideas. Can you provide me with a set of random words or phrases and allow me to brainstorm and develop my own ideas?"
Divergent Thinking:
- "Please provide me with a set of constraints for a design challenge, such as a limited color palette or a specific shape, and ask me to generate a list of unconventional solutions."
- "I'd like to solve a simple problem in an unconventional way. Can you provide me with a set of constraints and allow me to brainstorm and develop my own solution?"
Multi-Step Conversation:
- "Please provide me with a series of questions related to a given topic and ask me to provide step-by-step instructions for completing a task related to that topic."
- "I'd like to plan a trip to a new city. Can you provide me with a
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.