Skip to content
The Chat Archive
chatglm3-6b Arena Human Preference 55K

The Emergent Abilities of LLMs Could Be A Mirage! The best paper award in NeurIPs 2023…

  1. User

    The Emergent Abilities of LLMs Could Be A Mirage!

    The best paper award in NeurIPs 2023 went to a paper claiming that the emergent abilities of LLMs could be a mirage!

    The paper (link in alt) asserts that emergent abilities appear due to the researcher’s choice of metric rather than fundamental changes in model behavior with scale.

    Let's understand some terms before getting into the details.

    Emergence is a phenomenon whereby new properties may materialize in systems as their complexity increases. These properties can't be predicted from a precise quantitative understanding of the system’s microscopic details.

    Emergent properties of LLMs are abilities that are not present in small models that manifest themselves in larger models (Sharpness), and their performance on specific tasks can emerge quite unpredictably and abruptly at scale (Unpredictability)

    A lot of drama around LLMs taking over the planet involves emergence. Researchers argue that some scary emergent properties like free will and consciousness can magically manifest themselves in LLMS, and therefore, we have to pause, ban, and regulate AI research.

    The paper excellently and credibly argues that the LLMs DO NOT possess emergent abilities - by this, they mean that there isn't anything sharp or unpredictable about them.

    They show that smooth, continuous, predictable changes in model family performance appear sharp and unpredictable based on the choice of metric. So bigger models naturally and smoothly are more performant; there isn't some sharp jump in performance.

    They also find that for non-linear metrics, smaller models are more performant than previously reported when they add additional test points to increase the resolution of the benchmark.

    Overall, the point of the paper is that LLM behavior is NOT unpredictable, and in fact, larger LLMS are predictably more performant than larger ones.

    In other words, there is no scientific reason to believe that LLMs can magically become supervillains one day.

    If there is one paper you should read about LLMs, I recommend this one!

    TLDR: There is no magic voodoo happening with LLMs; it's all math and statistics... as all deep learning is

  2. Assistant

    The paper you've shared argues that the emergent abilities of large language models (LLMs) are not a result of the models having inherent emergent properties, but instead, are a result of the choices made by the researchers in terms of the metric used to evaluate their performance. The authors claim that by changing the metric used to evaluate these models, the emergent abilities can be made to disappear. Additionally, the paper argues that the performance of LLMs can be predicted and understood through a combination of math and statistics, and there is no evidence of these models having any supernatural or magical properties. Overall, this paper challenges the idea that LLMs are inherently unpredictable and uncontrollable, and instead, suggests that their behavior can be understood and predicted through scientific analysis.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.