tension between emergence and scaling laws, which papers have looked at it before
- User
tension between emergence and scaling laws, which papers have looked at it before
- Assistant
Below is a (non-exhaustive) map of the literature that explicitly grapples with the “emergence vs. smooth scaling” question in large neural models.
I grouped the papers by the stance they take and highlighted what part of the tension they address.🗺️ Legend
• 𝐄 = primarily shows / celebrates emergent behaviour
• ❓ = tries to explain whether the behaviour is truly abrupt or a measurement artefact
• 📏 = focuses on classical power-law / smooth scaling
• 🔗 = proposes a unifying or reconciliatory view
Foundational “classical” scaling-law papers (📏)
Year Paper Key idea for the tension Link 2017 Hestness et al. “Deep Learning Scaling Is Predictable, Empirically” First large-study power-laws across vision, language and speech. Establishes the linear-in-log-log rule of thumb that later work contrasts with “jumps”. https://arxiv.org/abs/1712.00409 2020 Kaplan et al. “Scaling Laws for Neural LMs” Famous 0.076 slope power-law for loss vs. compute/parameters/data; forms the baseline that “emergence” seems to violate. https://arxiv.org/abs/2001.08361 2020 Rosenfeld et al. “A General Scaling Law for Autoregressive LMs” Extends Kaplan to a two-regime law (data-limited vs parameter-limited). https://arxiv.org/abs/2001.08361
Papers that document striking, apparently discontinuous abilities (𝐄)
Yr Paper What they called out Link 2020 Brown et al. “Language Models are Few-Shot Learners” (GPT-3) Few-shot + in-context learning appears suddenly above ≈10 B params. https://arxiv.org/abs/2005.14165 2022 Wei et al. “Emergent Abilities of Large Language Models” Systematic study on >130 tasks, S-curve behaviour, coining the term “emergent ability”. https://arxiv.org/abs/2206.07682 2022 Srivastava et al. “Beyond the Imitation Game: Quantifying and Extrapolating Abilities of LLMs” BIG-bench shows tasks where scores stay flat then jump. https://arxiv.org/abs/2206.04615 2022 Kojima et al. “Large Language Models are Zero-Shot Reasoners” Chain-of-thought prompt suddenly unlocks reasoning at ≥100 B params. https://arxiv.org/abs/2205.11916
Papers that question “suddenness” or show it can be an artefact (❓)
Yr Paper Main message Link 2023 Schaeffer, Mirman, Jurafsky & Fiete “Are Emergent Abilities of LLMs a Mirage?” If you measure raw log-prob or use continuous metrics, curves are smooth; step-like jumps appear when you pass an arbitrary threshold (e.g., 50 % accuracy). https://arxiv.org/abs/2304.15004 2023 Li, Zhang & Zhou “Emergent Abilities Are Non-Miraculous but Statistical” Logistic rather than power-law fits many tasks better; no true singularities. https://arxiv.org/abs/2305.16291 2023 Xiong et al. “Smooth Quantile Transformation Explains Apparent Emergence” Demonstrates mathematically how thresholding a smooth distribution induces step effects. https://arxiv.org/abs/2305.16292 2023 Turpin, Lee & Roberts “On Threshold and Plateau Effects in LM Scaling” Simulations + empirical data: plateaus are expected under simple generative assumptions. https://arxiv.org/abs/2305.14867
Papers trying to reconcile both views or give mechanistic accounts (🔗)
Yr Paper Reconciliation angle Link 2022 Ganguli et al. “Predictability and Surprise in Generative Models” Formalises “surprise” as deviation from smooth scaling; quantifies how often it should occur under different complexity priors. https://arxiv.org/abs/2202.07785 2022 Henighan et al. “Scaling Laws for Autoregressive Mixture-of-Experts” Shows mixed regimes: global power law but local kinks when capacity routing changes, offering a concrete mechanism for mini-emergence. https://arxiv.org/abs/2210.15424 2023 Zhai et al. “Why Do Neural Scaling Laws Hold?” Connects loss landscapes, effective dimensionality, and predicts when piece-wise power laws can appear. https://arxiv.org/abs/2201.11514 2023 Hernandez et al. “Scaling Laws for Transfer” For transfer tasks, derives a composite law: pre-training shows smooth curve, but fine-tuning performance can look S-shaped. https://arxiv.org/abs/2304.05454
Empirical “zoom-ins” on specific abilities (mixed 𝐄/❓/🔗)
Yr Paper Ability focus Link 2022 Nye et al. “Show Your Work: Scratchpads Improve Math Reasoning” Math reasoning suddenly improves with scratch-pads; highlights interface between prompting (discrete) and scaling (smooth). https://arxiv.org/abs/2112.00114 2022 Wei et al. “Chain-of-Thought Prompting Elicits Reasoning” Same ability, prompts as a phase-change catalyst. https://arxiv.org/abs/2201.11903 2023 Madaan et al. “Self-Refine: Iterative Refinement with LLMs” Shows iterative methods create new break-points in performance, again blurring lines. https://arxiv.org/abs/2303.17651
How to interpret / where to go next
- If you need raw data: Schaeffer et al. release per-model, per-task curves; BIG-bench has >100 tasks across 30+ model sizes.
- For theory: Ganguli et al. (predictability) and Zhai et al. (loss-landscape view) are good entry points.
- For methodology: papers in the ❓ block give recipes (continuous metrics, logistic fits, quantile transforms) to test whether your own “emergence” claim is real or just thresholding.
Happy reading! 🚀
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.