Skip to content
The Chat Archive
o3-2025-04-16· OpenAI LMArena Human Preference 140K

How to do linear probing in Transformers?

  1. User

    How to do linear probing in Transformers?

  2. Assistant

    Below is a practical, end-to-end recipe for running a linear probe on a pretrained Transformer (e.g. BERT, RoBERTa, GPT-2) with PyTorch + 🤗 Transformers + scikit-learn.
    The same principles apply to vision models (ViT, CLIP, etc.); just swap the tokenizer / feature extractor.


    1. What is “linear probing”?

    1. Freeze the backbone network (all weights are kept fixed).
    2. Extract the hidden representation h from some layer ℓ (or several).
    3. Train a single linear layer (or logistic-regression classifier / ridge regressor) on top of those representations to predict a target label y.
    4. Evaluate. Good accuracy ⇒ the information you care about is linearly decodable at that layer.

    Why do this? • Inspect where linguistic / semantic information emerges in NLP models.
    • Compare models / pre-training objectives.
    • Check for memorization vs. abstraction.
    • Debug fine-tuning failures (“do I really need a full fine-tune?”).


    2. Set-up

    pip install transformers datasets scikit-learn torch tqdm
    

    3. Minimal code skeleton

    import torch, torch.nn as nn
    from transformers import AutoTokenizer, AutoModel
    from datasets import load_dataset
    from sklearn.linear_model import LogisticRegression
    from sklearn.metrics import accuracy_score
    from tqdm.auto import tqdm
    
    MODEL_NAME = "bert-base-uncased"
    LAYER_TO_PROBE = 8              # 0-11 for BERT-base
    BATCH_SIZE = 32
    MAX_LEN = 128                   # or shorter/longer if GPU allows
    
    # 1. Load model & tokenizer (frozen)
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
    model = AutoModel.from_pretrained(MODEL_NAME, output_hidden_states=True)
    model.eval()                    # disable dropout
    for p in model.parameters():
        p.requires_grad = False     # freeze
    
    # 2. Load / prepare your probing dataset
    # Example: "CoLA" grammatical acceptability classification (binary)
    dataset = load_dataset("glue", "cola")
    train, test = dataset["train"], dataset["validation"]
    
    def encode(batch):
        return tokenizer(batch["sentence"],
                         padding="max_length",
                         truncation=True,
                         max_length=MAX_LEN)
    
    train = train.map(encode, batched=True)
    test  = test .map(encode, batched=True)
    
    cols = ["input_ids", "attention_mask", "label"]
    train.set_format(type="torch", columns=cols)
    test .set_format(type="torch", columns=cols)
    
    # 3. Extract representations
    def get_representations(split):
        reps, labels = [], []
        loader = torch.utils.data.DataLoader(split, batch_size=BATCH_SIZE, shuffle=False)
        with torch.no_grad():
            for batch in tqdm(loader):
                out = model(input_ids=batch["input_ids"],
                            attention_mask=batch["attention_mask"])
                # out.hidden_states: tuple(layer0, layer1, …, layer12), each: [B, T, D]
                h = out.hidden_states[LAYER_TO_PROBE]          # choose layer ℓ
                cls = h[:, 0, :]                               # use [CLS] token
                reps.append(cls.cpu())
                labels.append(batch["label"].cpu())
        return torch.vstack(reps).numpy(), torch.cat(labels).numpy()
    
    X_train, y_train = get_representations(train)
    X_test , y_test  = get_representations(test)
    
    print("Rep shape:", X_train.shape)  # (N, hidden_size)
    
    # 4. Fit linear probe (LogReg with L2 acts like a single-layer classifier)
    probe = LogisticRegression(max_iter=1000, solver="lbfgs", C=1.0)
    probe.fit(X_train, y_train)
    
    # 5. Evaluate
    pred = probe.predict(X_test)
    acc = accuracy_score(y_test, pred)
    print(f"Layer {LAYER_TO_PROBE} accuracy: {acc:.3f}")
    

    4. Tips, variants & best practices

    1. Which token(s) to use
      • For sentence tasks with BERT-like models, [CLS] is standard.
      • Mean-pool or attention-weighted pool often works too.
      • For GPT-style (no CLS) you can use the last token or mean-pool.

    2. Multiple layers at once
      – Concatenate several layers (torch.cat([h3, h7, h11], dim=-1)).
      – Or train a separate probe per layer and plot accuracy vs. depth.

    3. Controlling capacity
      – Use ridge regression (sklearn.linear_model.RidgeClassifier) or a low C in LogisticRegression to keep the probe linear (avoid overfitting and leaking non-linear power).
      – Report random orientation baseline: shuffle labels, or use random features of the same dimensionality.

    4. Cross-validation
      – Do k-fold or dev-set tuning for hyper-parameters (C, penalty, class weight).
      – Use early stopping for large datasets (LogisticRegression has n_jobs=-1, max_iter).

    5. Statistical significance
      – Bootstrapping confidence intervals.
      – Run probes with multiple random seeds and report mean ± sd.

    6. Memory-friendly extraction
      – If GPU RAM is tight: send model to GPU but keep representations on CPU (reps.append(cls.cpu())).
      – Or write to disk as .npy / HF arrow file and fit the probe offline.

    7. Vision & audio models
      – Same idea; just grab the patch embeddings, average pool, etc.

    8. Libraries that automate probes
      – bertviz, probing (John Hewitt), dataclasses-json framework.
      – Adapter-Transformers has a “prediction head” option that can be frozen to behave like a probe.


    5. Interpreting results

    • Accuracy climbs → that information is encoded earlier than you thought.
    • Accuracy dips in middle layers → suggests mid-layers are re-packing information.
    • Higher performance ≠ causal importance. Combine with ablation or causal tracing techniques for stronger claims.


    🎉 That’s it! You now have a linear-probe pipeline to peek inside Transformer representations. Tweak the dataset, layer choice, pooling, and probe regularization to suit your research questions.

  3. User

    Why use logistic regression or train the probe with gradient descent? Why not use an SVM?

  4. Assistant

    Short answer ― you absolutely can use a linear SVM; it will give almost the same geometric test (“is the information linearly separable?”). In practice, most probing papers gravitate to logistic-regression / ridge because it is:

    1. Easier to train on millions of examples or 1000-dimensional hidden states.
    2. Naturally multi-class and gives probabilities.
    3. Smoothly differentiable, so you can optimise it with any off-the-shelf GPU SGD/Adam routine.
    4. Already built into every deep-learning library, which means one less dependency / solver to wrangle.

    Below is a more detailed comparison.


    1. Computational scalability (𝑁 = #examples, 𝐷 = hidden size)

    ModelCommon solverComplexity (rough)GPU-friendly?
    Logistic/RidgeLBFGS / (mini-batch) SGDO(N·D) per epochYes (pure matmul)
    Linear SVM (dual)LibLinear / SMO QPO(N² – N³) worst‐caseNo (CPU QP)
    Linear SVM (primal SGD)Pegasos / Hinge + SGDO(N·D) per epochYes, but needs custom loop

    If you have 500 k examples * 768-d vectors, a quadratic-time dual SVM may choke, while logistic regression trains in seconds with a single CUDA kernel.


    1. Smoothness of the loss

    • Logistic loss is smooth, so optimisers (LBFGS, Adam) get fast, steady convergence.

    • Hinge loss is piece-wise linear (non-differentiable at the margin). SGD works, but needs tricks (learning-rate decay, careful regularisation) to avoid oscillation.


    1. Probabilities & evaluation

    Probing papers often want: • class probabilities → entropy curves, AUC, calibration plots
    • multi-class tasks (POS tagging: 45 tags; sentiment: 5 levels)

    Softmax logistic regression supports this out-of-the-box.
    SVM needs one-vs-rest or Platt-scaling for probabilities, adding extra code and potential instability.


    1. Regularisation knobs

    Both models reduce to “linear weights + an ℓ2 penalty parameter C/λ”.
    Logistic regression’s single C hyper-parameter is identical in spirit to SVM’s, so switching rarely changes the story.
    Because results scarcely differ, authors pick the simpler tool.


    1. Interpretability

    With a probabilistic model you can interpret the weight vector as log-odds contributions, plot odds ratios layer-by-layer, etc.
    Hinge-loss weights have no probability semantics.


    1. Historical inertia

    Early NLP probing papers (Adi 2017, Hewitt & Manning 2019, Tenney 2019) used ridge / logistic. Subsequent work kept the convention for comparability.


    When might an SVM be preferable?

    • Very small datasets (few hundred points) where the max-margin property often generalises a bit better.
    • You explicitly care about support‐vector sparsity (e.g. counting #SVs as a capacity measure).
    • You’re probing binary tasks only and want deterministic hard margins.

    If that’s your case, by all means run a linear SVM:

    from sklearn.svm import LinearSVC
    probe = LinearSVC(C=1.0)        # hinge loss, no probs
    probe.fit(X_train, y_train)
    acc = probe.score(X_test, y_test)
    

    Or, for probabilities:

    from sklearn.svm import LinearSVC
    from sklearn.calibration import CalibratedClassifierCV
    svc  = LinearSVC(C=1.0)
    probe = CalibratedClassifierCV(svc)   # Platt scaling
    probe.fit(X_train, y_train)
    

    Bottom line
    • Logistic / ridge and linear SVM are functionally equivalent probes; both test linear separability.
    • Logistic regression is smoother, GPU-friendly, multi-class, probabilistic, and already tied into deep-learning stacks, so it wins the convenience race.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.