Role: PhD advisor, be helpful but very critical, the student is very capable, show where…
- User
Role: PhD advisor, be helpful but very critical, the student is very capable, show where the approach might hit pitfalls (and the essential difficulty to be resolved), do not propose a solution, just reveal the landscape of the problem
I'm thinking about linear attention or other state size constrained model and I want to train a model that can be trained at a smaller state size and then generalise to larger sizes. One example of the idea is random key attention where each training iteration has random keys. When we want to write a kv we do some sort of softmax based overwrite or meld of the values. The idea is that you can now change the key/memory size arbitrarily at inference time if you need a larger memory. I think the way to train it is to ensure that there is a distribution of memory sizes. For linear attention this is more complex as the state size is linked to the key dimension but key dimension is linked to a matrix multiply. Maybe we can use a rbf kernel and take the first n terms for a memory size of n so that the expanded key dimension equals the state size dimension. The thinking is that as we increase memory size, the gaps become smaller so that the network with 10000 memory and 10001 memory should be pretty much the same.
- Assistant
Here’s a brutally honest sketch of where this line of attack runs aground. I’ve broken it into the main conceptual hurdles—you’ll want to convince yourself you really understand each before diving deeper.
-
Conflating “memory-slots” with “key-vector dimension”
• In most transformer variants, the number of memory slots (N) and the dimensionality of each key (d) are orthogonal design choices. You seem to tie them together—sometimes treating them as interchangeable “state size.”
• If you train at d=64 and slice out d=32 at test time, you’re zeroing out half the parameters in your key/query projection. That dramatically shifts the function you’ve learned. Conversely, if you train at d=32 and then append 32 random, untrained dimensions at test time, you’ve introduced directions in feature space the network never saw.
→ Essential difficulty: achieving true invariance under changes in d requires an architecture whose parameterization doesn’t change shape or distribution when you add or remove dimensions. Standard linear layers do not satisfy that. -
Random-key attention and stability of retrieval
• If each training step re-samples keys, the network never learns consistent loci to “point to” in memory. You’re effectively asking it to retrieve by similarity in a totally shuffled embedding space every time.
• Overwrite/meld via a softmax over random anchors may converge to a trivial strategy: collapse onto the centroid of all values or ignore the addressing signal entirely.
→ Essential difficulty: you must balance two competing imperatives—(a) keys must be rich enough to discriminate different slots, (b) keys must be random enough to allow arbitrary enlargement at inference. These pull in opposite directions. -
Kernel-based linear attention and Mercer-term truncation
• Expanding an RBF (or any positive-definite) kernel in eigenfunctions or random Fourier features presumes a fixed basis. Truncating at term n changes the approximation error in a way that depends non-trivially on your data distribution and query distribution.
• As you increase n, you improve the approximation, but your network has to learn to produce query projections that remain well-behaved in the tail of that expansion. There’s no guarantee it will treat term 10001 as “almost the same” as term 10000—those higher-order basis functions can have wildly different spectral properties.
→ Essential difficulty: you’re relying on a smooth continuum of basis-function behavior—yet neural nets notoriously exploit sparsity and corner-case features rather than smooth interpolation. -
Distributional mismatch and generalization gaps
• Even if you train on a curriculum of memory sizes {32, 64, 128,…}, you’ll likely find interpolation at intermediate sizes works okay, but extrapolation (to much larger N or d) will degrade.
• The model can “cheat” by learning to focus on the first k slots (for some k seen at training) and ignore the rest. When you hand it 10× more slots, those extra slots might be treated as noise or swallowed by saturation in your overwrite mechanism.
→ Essential difficulty: there is no free lunch in function approximation—generalizing to a combinatorially larger configuration space (all possible memory sizes) without explicit architectural invariance is fundamentally at odds with how deep nets tend to latch onto narrow features.
Bottom line: you’re chasing a desideratum—“train once, run at arbitrary capacity”—that collides head-on with how parameterized linear maps and dot-product attention fundamentally work. Unless you discover (or constrain yourself to) a form of dimension-agnostic parameter sharing that provably commutes with slicing or padding, you will always bump into representational and stability cliffs.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.