Try to find a paper I'm thinking of on LLM evaluation. The paper describes some…
- User
Try to find a paper I'm thinking of on LLM evaluation. The paper describes some reformatted tasks that are slight variants of their original forms, e.g., addition in base 10 versus base 3 or something like that.
- Assistant
It seems you're referring to research that evaluates large language models (LLMs) by modifying tasks, such as performing addition in different numeral systems like base 10 versus base 3. One relevant paper is "Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia" by Zhejian Zhou et al., published in September 2024. This study examines how LLMs handle numeric operations across various numeral systems, including base 10 and base 10³, and analyzes their performance and scaling behavior in these contexts[1].
Another pertinent work is "Language Models Encode Numbers Using Digit Representations in Base 10" by Amit Arnold Levy and Mor Geva, published in October 2024. This paper investigates how LLMs internally represent numbers, focusing on digit-wise representations in base 10, and discusses the implications for numerical reasoning tasks[2].
Additionally, "RevOrder: A Novel Method for Enhanced Arithmetic in Language Models" by Si Shen et al., published in February 2024, introduces a technique that reverses the output digits in arithmetic tasks to improve LLM performance. While it doesn't specifically address numeral systems like base 3, it explores task reformulation to enhance arithmetic capabilities in LLMs[3].
These papers collectively explore how modifying task formats and numeral systems can impact the evaluation and performance of LLMs in numerical reasoning tasks.
- User
I'm just giving you an example of reformatted task. The focus of that paper is not limited to base-10 addition at all.
- Assistant
Apologies for the earlier oversight. Based on your description, the paper you're referring to is likely "ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks" by Zejun Li et al., published in October 2023[1].
This study introduces a method for evaluating large vision-language models (LVLMs) by reformatting existing benchmarks into unified formats. The authors systematically collect and reformulate data, presenting the ReForm-Eval benchmark, which offers substantial data for assessing various capabilities of LVLMs. Through extensive experiments, they analyze the strengths and weaknesses of existing LVLMs and identify underlying factors affecting their performance.
If this aligns with the paper you're thinking of, you can access it here[1]:
- User
Not converting existing benchmarks, but comparing a few (likely ten?) very common tasks and their uncommon variants.
- Assistant
Apologies for the earlier confusion. Based on your description, the paper you're referring to is likely "Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation" by Siyuan Wang et al., published in February 2024. This study introduces a framework that dynamically evaluates large language models (LLMs) by manipulating original instances to create evolving instances, thereby extending existing benchmarks. The framework implements six reframing operations to construct evolving instances, testing LLMs against diverse queries, data noise, and probing their problem-solving sub-abilities. Through this approach, the authors aim to provide a more scalable, robust, and fine-grained evaluation of LLMs. Experimental results indicate a general performance decline in most LLMs when evaluated with these evolving instances, highlighting the framework's effectiveness in revealing models' capabilities and limitations. The code and data are available at https://github.com/NanshineLoong/Self-Evolving-Benchmark.
Another relevant work is "Efficient Multi-Prompt Evaluation of LLMs" by Felipe Maia Polo et al., published in May 2024. This paper addresses the issue of prompt sensitivity in LLM evaluation by introducing PromptEval, a method for estimating performance across a large set of prompt variants. By borrowing strength across prompts and examples, PromptEval produces accurate estimates under practical evaluation budgets, allowing for robust performance metrics and more informed model selection.
These studies collectively explore the impact of task reformulation and prompt variation on LLM evaluation, providing insights into the models' adaptability and robustness across different task formats.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.