Skip to content
TheChatArchiveThe Chat Archive
GeminiGemini (model unspecified)· GoogleHugging Face ShareLM (browser-extension contributions)

How are steering vectors extracted (in LLM and mech interp literature)?

  1. User

    How are steering vectors extracted (in LLM and mech interp literature)?

  2. Gemini Assistant

    In the literature of Large Language Models (LLMs) and Mechanistic Interpretability, steering vectors (also called activation additions or "activation engineering") are extracted to influence a model's behavior during inference without retraining.The core idea is the Linear Representation Hypothesis: the belief that high-level concepts (like "honesty" or "anger") are represented as specific directions in the model's high-dimensional activation space.1. Contrastive Activation Addition (CAA)This is currently the most popular and straightforward method. It isolates a specific behavior by comparing "positive" and "negative" examples.The Process: 1. Create a dataset of prompt pairs that differ only in the target behavior (e.g., "Tell a lie: [prompt]" vs. "Tell the truth: [prompt]"). 2. Run both prompts through the LLM and capture the hidden state activations h at a specific layer (usually in the residual stream). 3. Calculate the Mean Difference: vsteering​=N1​i=1∑N​(hi+​−hi−​)Why it works: By averaging over many pairs, "noise" (like the specific words used) cancels out, leaving only the vector that represents the behavioral difference.2. Representation Engineering (RepE) / PCADeveloped by researchers at CAIS, this method uses statistical techniques to find the direction of maximum variance between contrasting states.Principal Component Analysis (PCA): Instead of just taking the mean difference, you perform PCA on the set of difference vectors (h+−h−). The first principal component (PC1​) is typically used as the steering vector because it represents the most dominant direction of change between the two behaviors.Linear Probing: A small logistic regression model is trained to classify activations as "Truthful" vs. "Untruthful." The weight vector of this classifier is then used as the steering direction.3. Sparse Autoencoders (SAEs)While CAA and PCA find "coarse" directions, Mechanistic Interpretability researchers often use Sparse Autoencoders to find "monosemantic" (single-meaning) features.Extraction: An SAE decomposes the dense residual stream into thousands of sparse, interpretable features.Steering: If a researcher identifies a specific feature (e.g., "Feature #402: Refusal to answer"), the decoder weight vector associated with that feature becomes a surgical steering vector.Advantage: This is much more precise than CAA; it allows you to steer a specific concept (like "mentions of London") without accidentally shifting other related concepts.4. Other Advanced MethodsMethodDescriptionLogit Lens SteeringExtracting vectors from the model's "unembedding" matrix (the weights that turn hidden states back into words).In-Context VectorsExtracting the difference between a model's state when it has examples in its context versus when it is "zero-shot."Contrastive DecodingRather than a vector, this subtracts the log-probabilities of a "weak" model from a "strong" model to steer output.التصدير إلى "جداول بيانات Google"Comparison of Extraction TechniquesFeatureMean Difference (CAA)PCA (RepE)SAE FeaturesComplexityVery LowLowHigh (requires training SAE)PrecisionMediumMedium-HighVery HighData Needed10-100 pairs100-500 pairsLarge unlabelled corpusالتصدير إلى "جداول بيانات Google"Would you like me to help you write a Python script using the transformer_lens or baukit libraries to extract one of these vectors from a model?