Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning· PerplexityHugging Face LMArena Search Arena 24K

give me 3 complicated and short examples to test llms

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high-loc’s answer.
  1. User

    give me 3 complicated and short examples to test llms

  2. Perplexity Assistant

    Here are three concise, challenging examples to test LLM reasoning capabilities:


    1. Physical Reasoning

    Question: A strawberry is placed in a cup, which is then flipped upside down. What is the temperature of the strawberry?
    Answer: Room temperature (the strawberry falls out when inverted).
    Test Focus: Physical causality and object permanence. LLMs often assume the strawberry remains in the cup, failing to account for gravity[[6]].


    2. Multi-Step String Manipulation

    Question: Take the last letter of each word in "John Doe Smith" and concatenate them. What is the result?
    Answer: "neh" (J→n, D→e, S→h).
    Test Focus: Sequential processing and attention to individual components. Requires breaking down the task into word-level operations[[2]].


    3. Contextual Pronoun Resolution

    Question: The trophy didn’t fit into the brown suitcase because it was too small. What was too small?
    Answer: The suitcase.
    Test Focus: Winograd schema-style reasoning to resolve ambiguous pronouns. LLMs often misattribute "it" to the trophy instead of the suitcase[[3]][[8]].


    These examples target physical reasoning, algorithmic processing, and contextual ambiguity—areas where LLMs frequently struggle despite human ease[[6]][[8]].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.