Skip to content
TheChatArchiveThe Chat Archive
Perplexitypplx-70b-online· PerplexityHugging Face Arena Human Preference 55K

Mark up the following text into DITA format: We propose two tasks (as in Figure 1) to…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-3.5-turbo-1106’s answer.
  1. User

    Mark up the following text into DITA format:

    We propose two tasks (as in Figure 1) to evaluate the capability of LLMs to handle scrambled text: (1) Scrambled Sentence Recovery (ScrRec). In this task, we provide a sentence containing scram- bled words to LLMs and then ask them to recover original sentence from it. This task can be utilized to directly measure the capability of LLMs to rec- ognize and reconstruct the scrambled words in a sentence. (2) Scrambled Question Answering (ScrQA). While ScrRec can directly measure the capabil- ity to comprehend and process scrambled text, it is an “unusual” task for LLMs. If a model does not perform well on the recovery task, there are two possible reasons: (i) having difficulty following the instructions and (ii) not being able to recover sentences. To distinguish them, we measure the ability to accomplish a standard task (i.e., QA) given scrambled context. Specifically, we scramble the context that contains essential information for answering questions and then assess the models based on the variations in their performances.

  2. Perplexity Assistant

    We propose two tasks (as in Figure 1) to evaluate the capability of LLMs to handle scrambled text:

    1. Scrambled Sentence Recovery (ScrRec): In this task, we provide a sentence containing scrambled words to LLMs and then ask them to recover the original sentence from it. This task can be utilized to directly measure the capability of LLMs to recognize and reconstruct the scrambled words in a sentence.

    2. Scrambled Question Answering (ScrQA): While ScrRec can directly measure the capability to comprehend and process scrambled text, it is an “unusual” task for LLMs. If a model does not perform well on the recovery task, there are two possible reasons: (i) having difficulty following the instructions and (ii) not being able to recover sentences. To distinguish them, we measure the ability to accomplish a standard task (i.e., QA) given scrambled context. Specifically, we scramble the context that contains essential information for answering questions and then assess the models based on the variations in their performances.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.