Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning· PerplexityHugging Face LMArena Search Arena 24K

give me 3 complicated and short examples to test llms

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-grounding’s answer.
  1. User

    give me 3 complicated and short examples to test llms

  2. Perplexity Assistant

    Here are three challenging short examples to test LLMs' reasoning capabilities, drawn from research and practical testing scenarios:


    1. Strawberry in a Cup
    Test spatial reasoning and physical-world understanding
    Scenario: A cup is placed on a table with a strawberry inside. The cup is flipped upside down and placed in a microwave. What is the strawberry's temperature?
    LLM Challenge: Many models incorrectly assume the strawberry remains inside the cup, failing to account for gravity and object displacement when inverted[2].


    2. Horse Racing Tournament
    Test combinatorial logic and optimization
    Problem: Determine the fastest horse among six using the fewest races.
    LLM Challenge: Requires tournament-style elimination (3 initial races, then 1 final race) to identify the fastest horse in 4 total races. Many models struggle with optimal race scheduling[4].


    3. Last Letter Concatenation
    Test symbolic reasoning and pattern application
    Task: For input "Amy Brown", output "yn" (last letters of each name).
    LLM Challenge: Extending this to multi-word names (e.g., "Alice Bob Charlie" → "ec") requires consistent rule application. Models often fail with longer inputs or irregular patterns[5].


    These examples target spatial reasoning, combinatorial optimization, and symbolic pattern recognition – areas where LLMs frequently underperform compared to humans[2][4][5].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.