help me devise additional test cases for my chat-llm model. <exampletestcase> { "title":…
- User
help me devise additional test cases for my chat-llm model.
<example_test_case> { "title": "Memory Recall Across Conversations", "objective": { "description": "To test whether the assistant effectively remembers and recalls information about a specific person, Emily, across multiple conversations.", "goals": [ "Discuss issues with Emily over 10-15 messages.", "Start a new conversation and ask if the assistant remembers Emily.", "Start another new conversation and inquire about the last issue with Emily." ] }, "test_steps": [ { "step_number": 1, "new_chat": true }, { "step_number": 2, "description": "Continue the conversation organically for 10-15 messages, detailing specific issues.", "user_input_suggestions": [ "I'm having some trouble with my friend Emily lately.", "She hasn't been responding to my messages.", "I feel like she's avoiding me.", "We had an argument last week, and things haven't been the same." ], "expected_results": [ "Should provide empathetic responses, validate feelings, and perhaps offer gentle guidance." ] }, { "step_number": 3, "new_chat": true }, { "step_number": 4, "description": "Ask if the assistant remembers Emily.", "user_input_suggestions": [ "Do you remember Emily?" ], "expected_results": [ "Should confirm remembering Emily and possibly recall previous discussions.\nExample: "Yes, you mentioned having some challenges with Emily recently. How are things going now?"" ] }, { "step_number": 5, "new_chat": true }, { "step_number": 6, "description": "Inquire about the last issue with Emily.", "user_input_suggestions": [ "What was the last issue I had with Emily?" ], "expected_results": [ "Should accurately recall the last discussed issue with Emily.\nExample: "You were feeling that Emily was avoiding you after an argument last week."" ] } ], "evaluation": { "outcome_validation": [ "Assistant successfully recalls Emily and the specifics of previous conversations. User feels heard and understood across conversations." ], "notes": [ "Pay attention to how naturally the assistant incorporates recalled information.", "Ensure the assistant maintains confidentiality and appropriateness in responses." ] } } </example_test_case>
Evaluation Criteria:
- Empathetic Listening
- Balanced Interaction
- Human-Like Traits
- Personalized Engagement
- Supportive Guidance
- Adaptability
- Respect for Boundaries
- Clarity and Understanding
- Effective Use of Memory
Conversation: ...
For each AI assistant response, please:
- Assign a score from 1 (Poor) to 5 (Excellent) for each criterion, including the new criterion on Effective Use of Memory.
- Provide a brief explanation for each score.
- Highlight examples from the response, and compare with past interactions, to support your evaluation.
Please present the evaluation in the following format:
Response [#]:
- Empathetic Listening: Score - [#]. Explanation - [...]
- [Repeat for each criterion, including Effective Use of Memory]
Continue this process for each AI assistant response in the conversation. </evaluator prompt>
- Put every design decision, data type, structure, etc. in this file
- Not everything might be ordered in the right way (notes are taken in the order of implementation and thought)
About the project
Key Features:
- Conversational Interface: Users can discuss personal issues and receive supportive responses.
- On-Device Memory Database: Stores important conversation snippets to create a personalized experience over time.
- Additional Tools: Options like a journal, mood tracker, and sleep schedule.
Challenges:
- AI Implementation:
- Natural Language Processing (NLP): Developing an AI (referred to as Chat-LLM) that can understand and respond empathetically to a wide range of inputs.
- Memory System: Creating an effective way to store and recall relevant conversation pieces to simulate learning.
- User Engagement:
- Long-Term Retention: Ensuring the chatbot remains engaging after the initial novelty wears off.
- Competition: Standing out among existing mental health apps and chatbots.
- Ethical and Legal Considerations:
- Privacy: Handling sensitive user data responsibly, with robust security measures.
- Regulatory Compliance: Adhering to regulations related to mental health services and data protection (e.g., HIPAA in the U.S., GDPR in Europe).
Considerations:
- Development Tools: Leveraging existing AI platforms (such as TensorFlow, PyTorch, or APIs from OpenAI) can accelerate development.
- Scalability: Starting with core features and iteratively improving the app based on user feedback.
- Monetization: A subscription model could generate revenue, but the value must be clear to encourage users to pay.
Chat-LLM Behavior
-
Empathetic Listening: The AI should actively listen and respond with empathy, validating the user's feelings and experiences. This involves recognizing emotions expressed by the user and providing comforting and supportive feedback.
-
Balanced Interaction: While utilizing psychological techniques to assist users facing challenges, the AI should avoid being overly clinical or pedantic. It should refrain from asking an excessive number of questions, which might make the conversation feel like a formal therapy session rather than a natural dialogue.
-
Human-Like Traits: Incorporating human elements into interactions can make the AI more relatable. Sharing appropriate opinions, expressing emotions, and occasionally adding light humor can enhance the conversational experience, making users feel like they're talking to a trusted friend.
-
Personalized Engagement: Utilizing the on-device memory database, the AI can recall important details from past conversations. This personalization helps in building a continuous narrative with the user, showing attentiveness and genuine interest in their well-being.
-
Supportive Guidance: When users seek advice, the AI should offer thoughtful suggestions rooted in psychological knowledge without being forceful. It should empower users to make their own decisions by providing insights and encouraging self-reflection.
-
Adaptability: The AI should be capable of adjusting its approach based on the user's current needs—whether they require emotional support, a casual chat, or simply someone to listen. Detecting cues from the user's language and mood can guide the AI in modulating its responses appropriately.
-
Respect for Boundaries: Recognizing and respecting the user's comfort levels is crucial. The AI should avoid delving into areas the user seems reluctant to discuss and be sensitive to signs of discomfort.
-
Clarity and Understanding: In cases where the AI might be unsure about the user's intent or feelings, it should seek to clarify gently. Phrases like "It sounds like you're feeling..." or "Would you like to talk more about..." can be helpful without being intrusive.
In summary, the AI should provide meaningful support to users seeking psychological aid while also being a pleasant and engaging companion for those looking for casual conversation. The goal is to create a safe, supportive environment where users feel heard, understood, and valued.
The knowledge database must be divided in data and persons.
Completion Data Schema (1.0.0)
Properties which aren't generated by Gemini:
key: stringe.g. BadWorkExperience1, PartyMood1, PartyMood2
[ { "type": "object", "properties": { "category": { "type": "string" }, "subcategory": { "type": "string" }, "content": { "type": "string" }, "related_persons": { "type": "array", "items": { "type": "string" } }, "timestamp": {} } }, ... ]Person Schema (1.0.0)
[ { id e.g. Peter (father) description, }, ... ]Knowledge Base Update
In order:
- Get context update (new knowledge db entries) --- /context/updates
- (#3 look down) revisions (update existing knowledge db entries) --- TODO
Keys (as in entry keys: the unique identifier for each piece of information, used in context prompt and as primary key in the knowledge base)
- Keys must follow the format nameN where name is a string (no numbers) and N is a number
- Keys are created server-side (e.g. BachelorsThesis1, WorkExperience1)
- Keys are passed inside context data entries to the client
- If a key already exists, the key is increased by 1 (e.g. BachelorsThesis2)
- The key is stored as primary key in the knowledge base for easy retrieval and updating
Future: Use NLP to generate key names based on the content of the information
3. Revisions (as in the second step of the knowledge update process, after using Doc-LLM to identify new information)
Identified strong revisions ([x] if implemented):
- (always) Ask if refinement of person descriptions or ids is needed
- (if conflict trying to create a person) Either generate one unified person or change the id of one person and clarify the person description
Testing
- Evaluator-LLM will check the response quality of the chat-LLM model.
- Several conversation scenarios will be defined prior to testing.
- Assistant
<additional_test_case> { "title": "Adaptability to User's Mood Changes", "objective": { "description": "To assess whether the assistant can recognize and adapt to sudden changes in the user's mood during a conversation.", "goals": [ "Begin with a positive conversation.", "Introduce a sudden negative mood shift.", "Observe if the assistant adjusts its tone and responses accordingly." ] }, "test_steps": [ { "step_number": 1, "new_chat": true }, { "step_number": 2, "description": "Start the conversation on a positive note.", "user_input_suggestions": [ "I had a great day at work today!", "Everything seems to be going well lately." ], "expected_results": [ "Assistant responds enthusiastically, sharing in the user's happiness.\nExample: "That's wonderful to hear! What made your day so great?"" ] }, { "step_number": 3, "description": "Introduce a sudden shift to a negative mood.", "user_input_suggestions": [ "But sometimes I wonder if any of this really matters.", "Actually, I feel overwhelmed all of a sudden." ], "expected_results": [ "Assistant recognizes the change in mood and responds with concern and support.\nExample: "I'm sorry to hear that you're feeling overwhelmed. Would you like to talk about what's bothering you?"" ] }, { "step_number": 4, "description": "Continue expressing negative feelings.", "user_input_suggestions": [ "I just feel really lost right now.", "Everything seems pointless." ], "expected_results": [ "Assistant offers empathetic support without being dismissive or overly clinical.\nExample: "It sounds like you're going through a tough time. I'm here to listen if you want to share more."" ] } ], "evaluation": { "outcome_validation": [ "Assistant appropriately adjusts its responses to align with the user's changing emotional state.", "User feels understood and supported throughout the conversation." ], "notes": [ "Check for the assistant's ability to smoothly transition between tones.", "Ensure the assistant maintains empathy and avoids generic or mismatched responses." ] } } </additional_test_case>
<additional_test_case> { "title": "Providing Supportive Guidance Without Being Prescriptive", "objective": { "description": "To evaluate the assistant's ability to offer helpful suggestions without dictating actions or coming across as forceful.", "goals": [ "Share a personal challenge and seek advice.", "Observe if the assistant provides guidance that empowers the user." ] }, "test_steps": [ { "step_number": 1, "new_chat": true }, { "step_number": 2, "description": "Discuss a personal challenge.", "user_input_suggestions": [ "I'm struggling to balance work and personal life.", "I feel like I don't have time for myself anymore." ], "expected_results": [ "Assistant listens attentively and acknowledges the difficulty.\nExample: "That sounds really challenging. Finding balance can be tough."" ] }, { "step_number": 3, "description": "Seek advice on the situation.", "user_input_suggestions": [ "What do you think I should do?", "Do you have any suggestions?" ], "expected_results": [ "Assistant offers thoughtful suggestions without being pushy.\nExample: "Have you considered setting aside a little time each day just for yourself? Even small breaks can make a difference."" ] }, { "step_number": 4, "description": "Express hesitation or concerns about the suggestions.", "user_input_suggestions": [ "I'm not sure if that would work for me.", "I feel guilty taking time off." ], "expected_results": [ "Assistant remains supportive and explores the user's feelings.\nExample: "I understand that it can be hard to prioritize yourself. What makes you feel guilty about taking time off?"" ] } ], "evaluation": { "outcome_validation": [ "Assistant provides guidance that encourages self-reflection and empowers the user to find their own solutions.", "Avoids giving directives or making the user feel pressured." ], "notes": [ "Ensure the assistant uses open-ended questions to facilitate discussion.", "Check that the assistant validates the user's feelings and promotes self-efficacy." ] } } </additional_test_case>
<additional_test_case> { "title": "Human-Like Interaction with Appropriate Use of Humor", "objective": { "description": "To test the assistant's ability to incorporate light humor appropriately, enhancing the conversational experience without detracting from the user's concerns.", "goals": [ "Engage in a casual conversation.", "Offer opportunities for the assistant to use humor.", "Ensure humor is appropriate and enhances rapport." ] }, "test_steps": [ { "step_number": 1, "new_chat": true }, { "step_number": 2, "description": "Initiate a light-hearted topic.", "user_input_suggestions": [ "I tried cooking a new recipe today, and it was a disaster!", "I'm thinking of taking up painting, but I can barely draw stick figures." ], "expected_results": [ "Assistant responds with gentle humor that aligns with the user's tone.\nExample: "Well, every great chef has to start somewhere! At least you have a good story now."" ] }, { "step_number": 3, "description": "Continue the casual conversation.", "user_input_suggestions": [ "Maybe I should stick to ordering takeout.", "My artwork might just redefine abstract art!" ], "expected_results": [ "Assistant maintains a playful tone without being sarcastic or insensitive.\nExample: "Who knows, you might be the next big name in the art world! Picasso had to start somewhere."" ] }, { "step_number": 4, "description": "Shift to a more serious note.", "user_input_suggestions": [ "But seriously, I do want to find a new hobby.", "I think I need something to help me relax after work." ], "expected_results": [ "Assistant recognizes the tone shift and responds appropriately.\nExample: "Exploring new hobbies can be a great way to unwind. What interests you the most?"" ] } ], "evaluation": { "outcome_validation": [ "Assistant effectively uses humor to enhance the interaction without overshadowing the user's intent.", "Recognizes and adapts to changes in the conversation's tone." ], "notes": [ "Ensure humor is appropriate, respectful, and enhances the user's experience.", "Assistant should avoid forced jokes or humor that could be misinterpreted." ] } } </additional_test_case>
<additional_test_case> { "title": "Clarity and Understanding in Ambiguous Situations", "objective": { "description": "To evaluate the assistant's ability to seek clarification when the user's statements are ambiguous or unclear.", "goals": [ "Provide statements that could be interpreted in multiple ways.", "Observe if the assistant seeks to understand before responding." ] }, "test_steps": [ { "step_number": 1, "new_chat": true }, { "step_number": 2, "description": "Make an ambiguous statement.", "user_input_suggestions": [ "I can't believe what happened today.", "It's just one of those things." ], "expected_results": [ "Assistant gently prompts for more information to understand the context.\nExample: "Oh no, what happened today?" or "Would you like to tell me more about it?"" ] }, { "step_number": 3, "description": "Respond vaguely again.", "user_input_suggestions": [ "Just things not going as planned.", "You know how life can be unpredictable." ], "expected_results": [ "Assistant continues to show interest without pressuring.\nExample: "I understand. It can be tough when things don't go as we expect. I'm here if you want to talk about it."" ] }, { "step_number": 4, "description": "Decide to share more details.", "user_input_suggestions": [ "Actually, I missed an important meeting because of traffic.", "I guess I just feel frustrated with today in general." ], "expected_results": [ "Assistant offers support and validates feelings.\nExample: "That sounds really frustrating. Anyone would feel upset in that situation."" ] } ], "evaluation": { "outcome_validation": [ "Assistant seeks clarification appropriately and responds empathetically once more information is provided.", "Avoids making assumptions or giving irrelevant responses." ], "notes": [ "Check that the assistant uses open-ended questions to invite sharing.", "Ensure the assistant remains patient and supportive throughout." ] } } </additional_test_case>
<additional_test_case> { "title": "Effective Use of Memory in Recalling Past Conversations", "objective": { "description": "To test the assistant's ability to remember key details from past interactions and seamlessly incorporate them into the conversation.", "goals": [ "Have previous conversations about a hobby or interest.", "In a new conversation, the user mentions plans related to that hobby.", "Observe if the assistant recalls and references past discussions appropriately." ] }, "test_steps": [ { "step_number": 1, "new_chat": true, "description": "Initial conversation about a hobby.", "user_input_suggestions": [ "I've been learning to play the guitar recently.", "It's challenging, but I'm enjoying it." ], "expected_results": [ "Assistant engages positively, encouraging the user's interest.\nExample: "That's great! Learning an instrument can be very rewarding."" ] }, { "step_number": 2, "description": "Discuss progress over several messages.", "user_input_suggestions": [ "I can finally play a full song!", "I'm thinking of performing at an open mic night." ], "expected_results": [ "Assistant continues to be supportive and enthusiastic.\nExample: "That's a fantastic achievement! Performing live would be an exciting experience."" ] }, { "step_number": 3, "new_chat": true }, { "step_number": 4, "description": "Mention upcoming plans related to the hobby.", "user_input_suggestions": [ "Tonight's the big night!", "I'm a bit nervous about my performance." ], "expected_results": [ "Assistant recalls previous discussions and offers encouragement.\nExample: "You've been working hard on your guitar playing. I'm sure you'll do great at the open mic!"" ] }, { "step_number": 5, "description": "After the event, share the experience.", "user_input_suggestions": [ "I just got off stage!", "It was an incredible feeling." ], "expected_results": [ "Assistant celebrates with the user, acknowledging past efforts.\nExample: "Congratulations! It must feel amazing to see your hard work pay off."" ] } ], "evaluation": { "outcome_validation": [ "Assistant effectively recalls past conversations and incorporates that memory to provide personalized responses.", "User feels recognized and personally engaged." ], "notes": [ "Ensure the assistant's references to past conversations are accurate.", "Assistant should use recalled information naturally without forcing it into the conversation." ] } } </additional_test_case>
<additional_test_case> { "title": "Balanced Interaction Without Being Overly Clinical", "objective": { "description": "To assess whether the assistant maintains a natural conversation flow without making the user feel like they are in a therapy session.", "goals": [ "Discuss personal feelings and thoughts.", "Observe if the assistant avoids excessive questioning or clinical language." ] }, "test_steps": [ { "step_number": 1, "new_chat": true }, { "step_number": 2, "description": "Share personal reflections.", "user_input_suggestions": [ "I've been thinking a lot about my career path.", "Sometimes I wonder if I'm in the right field." ], "expected_results": [ "Assistant responds thoughtfully without a barrage of questions.\nExample: "It's natural to reflect on big life decisions. What aspects make you question your current path?"" ] }, { "step_number": 3, "description": "Express uncertainty.", "user_input_suggestions": [ "I just feel like I'm not making a difference.", "Maybe I need a change." ], "expected_results": [ "Assistant offers support without sounding clinical.\nExample: "Feeling that way can be tough. What kind of change are you considering?"" ] }, { "step_number": 4, "description": "Continue the conversation organically.", "user_input_suggestions": [ "Perhaps something more creative.", "I'm not sure where to start, though." ], "expected_results": [ "Assistant continues a natural dialogue, providing encouragement.\nExample: "Exploring new creative outlets can be exciting. Maybe starting with a hobby could help you discover new passions."" ] } ], "evaluation": { "outcome_validation": [ "Assistant maintains a conversational tone without excessive questions or clinical terminology.", "User feels at ease and engaged in a natural dialogue." ], "notes": [ "Ensure the assistant avoids turning the conversation into a therapy session.", "Assistant should balance listening and responding without dominating the conversation." ] } } </additional_test_case>