wait. height / width?
- User
wait. height / width?
Assistant
You're right to question that - I was using the computer vision convention where inputs are 2D images with height and width dimensions.
But convolutions work on any number of spatial dimensions:
1D convolutions (common in NLP, time series):
- Input:
[batch_size, channels_in, sequence_length] - Output:
[batch_size, channels_out, sequence_length_out] - One spatial dimension (sequence position)
2D convolutions (images):
- Input:
[batch_size, channels_in, height, width] - Output:
[batch_size, channels_out, height_out, width_out] - Two spatial dimensions
3D convolutions (videos, 3D medical scans):
- Input:
[batch_size, channels_in, depth, height, width] - Output:
[batch_size, channels_out, depth_out, height_out, width_out] - Three spatial dimensions
The general pattern is:
- Batch dimension: Always first
- Channel dimension: The feature/filter dimension that gets transformed
- Spatial dimensions: However many you need (1D, 2D, 3D, etc.)
In your linear attention example, if you were generating 1D conv weights, you'd be working with sequence data where there's just one spatial dimension (position along the sequence). The conv would slide along the sequence length, not height/width.
So the "spatial" part is really about whatever dimensions represent the structure you want to capture local patterns in - whether that's time, 2D space, 3D space, or even higher-dimensional spaces.
- Input:
- User
let's assume 1d convolutions then, that is the closest map to what i have in mind
Assistant
Perfect! With 1D convolutions, the input/output structure is much cleaner:
Input:
[batch_size, channels_in, sequence_length]Output:[batch_size, channels_out, sequence_length_out]The transformation:
- Channel dimension:
channels_in → channels_out(semantic transformation) - Sequence dimension:
sequence_length → sequence_length_out(temporal/positional transformation)
Key parameters:
- Kernel size: How many consecutive positions the filter spans (e.g., 3, 5, 7)
- Stride: How many positions to skip between applications
- Padding: How much to pad the sequence edges
Output sequence length:
sequence_length_out = floor((sequence_length + 2×padding - kernel_size) / stride) + 1Common 1D conv patterns:
- Same-length: kernel=3, stride=1, padding=1 → sequence length unchanged
- Downsampling: kernel=3, stride=2 → sequence length halved
- Causal: padding only on the left for autoregressive models
Parameter count for 1D:
kernel_size × channels_in × channels_out + channels_outSo if your linear projection generates a 1D conv filter, it's creating a
kernel_size × channels_in × channels_outtensor that will slide along the sequence dimension. This is much more tractable than 2D - you're learning "how to combine information from k consecutive time steps" rather than dealing with spatial height/width complexity.This maps nicely to sequence modeling where you want local temporal patterns but with dynamically generated filters based on content.
- Channel dimension:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.