Skip to content
The Chat Archive
o3-2025-04-16· OpenAI LMArena Human Preference 140K

tension between emergence and scaling laws, which papers have looked at it before

  1. User

    tension between emergence and scaling laws, which papers have looked at it before

  2. Assistant

    Below is a (non-exhaustive) map of the literature that explicitly grapples with the “emergence vs. smooth scaling” question in large neural models.
    I grouped the papers by the stance they take and highlighted what part of the tension they address.

    🗺️ Legend
    • 𝐄 = primarily shows / celebrates emergent behaviour
    • ❓ = tries to explain whether the behaviour is truly abrupt or a measurement artefact
    • 📏 = focuses on classical power-law / smooth scaling
    • 🔗 = proposes a unifying or reconciliatory view


    Foundational “classical” scaling-law papers (📏)

    YearPaperKey idea for the tensionLink
    2017Hestness et al. “Deep Learning Scaling Is Predictable, Empirically”First large-study power-laws across vision, language and speech. Establishes the linear-in-log-log rule of thumb that later work contrasts with “jumps”.https://arxiv.org/abs/1712.00409
    2020Kaplan et al. “Scaling Laws for Neural LMs”Famous 0.076 slope power-law for loss vs. compute/parameters/data; forms the baseline that “emergence” seems to violate.https://arxiv.org/abs/2001.08361
    2020Rosenfeld et al. “A General Scaling Law for Autoregressive LMs”Extends Kaplan to a two-regime law (data-limited vs parameter-limited).https://arxiv.org/abs/2001.08361

    Papers that document striking, apparently discontinuous abilities (𝐄)

    YrPaperWhat they called outLink
    2020Brown et al. “Language Models are Few-Shot Learners” (GPT-3)Few-shot + in-context learning appears suddenly above ≈10 B params.https://arxiv.org/abs/2005.14165
    2022Wei et al. “Emergent Abilities of Large Language Models”Systematic study on >130 tasks, S-curve behaviour, coining the term “emergent ability”.https://arxiv.org/abs/2206.07682
    2022Srivastava et al. “Beyond the Imitation Game: Quantifying and Extrapolating Abilities of LLMs”BIG-bench shows tasks where scores stay flat then jump.https://arxiv.org/abs/2206.04615
    2022Kojima et al. “Large Language Models are Zero-Shot Reasoners”Chain-of-thought prompt suddenly unlocks reasoning at ≥100 B params.https://arxiv.org/abs/2205.11916

    Papers that question “suddenness” or show it can be an artefact (❓)

    YrPaperMain messageLink
    2023Schaeffer, Mirman, Jurafsky & Fiete “Are Emergent Abilities of LLMs a Mirage?”If you measure raw log-prob or use continuous metrics, curves are smooth; step-like jumps appear when you pass an arbitrary threshold (e.g., 50 % accuracy).https://arxiv.org/abs/2304.15004
    2023Li, Zhang & Zhou “Emergent Abilities Are Non-Miraculous but Statistical”Logistic rather than power-law fits many tasks better; no true singularities.https://arxiv.org/abs/2305.16291
    2023Xiong et al. “Smooth Quantile Transformation Explains Apparent Emergence”Demonstrates mathematically how thresholding a smooth distribution induces step effects.https://arxiv.org/abs/2305.16292
    2023Turpin, Lee & Roberts “On Threshold and Plateau Effects in LM Scaling”Simulations + empirical data: plateaus are expected under simple generative assumptions.https://arxiv.org/abs/2305.14867

    Papers trying to reconcile both views or give mechanistic accounts (🔗)

    YrPaperReconciliation angleLink
    2022Ganguli et al. “Predictability and Surprise in Generative Models”Formalises “surprise” as deviation from smooth scaling; quantifies how often it should occur under different complexity priors.https://arxiv.org/abs/2202.07785
    2022Henighan et al. “Scaling Laws for Autoregressive Mixture-of-Experts”Shows mixed regimes: global power law but local kinks when capacity routing changes, offering a concrete mechanism for mini-emergence.https://arxiv.org/abs/2210.15424
    2023Zhai et al. “Why Do Neural Scaling Laws Hold?”Connects loss landscapes, effective dimensionality, and predicts when piece-wise power laws can appear.https://arxiv.org/abs/2201.11514
    2023Hernandez et al. “Scaling Laws for Transfer”For transfer tasks, derives a composite law: pre-training shows smooth curve, but fine-tuning performance can look S-shaped.https://arxiv.org/abs/2304.05454

    Empirical “zoom-ins” on specific abilities (mixed 𝐄/❓/🔗)

    YrPaperAbility focusLink
    2022Nye et al. “Show Your Work: Scratchpads Improve Math Reasoning”Math reasoning suddenly improves with scratch-pads; highlights interface between prompting (discrete) and scaling (smooth).https://arxiv.org/abs/2112.00114
    2022Wei et al. “Chain-of-Thought Prompting Elicits Reasoning”Same ability, prompts as a phase-change catalyst.https://arxiv.org/abs/2201.11903
    2023Madaan et al. “Self-Refine: Iterative Refinement with LLMs”Shows iterative methods create new break-points in performance, again blurring lines.https://arxiv.org/abs/2303.17651

    How to interpret / where to go next

    1. If you need raw data: Schaeffer et al. release per-model, per-task curves; BIG-bench has >100 tasks across 30+ model sizes.
    2. For theory: Ganguli et al. (predictability) and Zhai et al. (loss-landscape view) are good entry points.
    3. For methodology: papers in the ❓ block give recipes (continuous metrics, logistic fits, quantile transforms) to test whether your own “emergence” claim is real or just thresholding.

    Happy reading! 🚀

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.