Skip to content
The Chat Archive
o3-2025-04-16· OpenAI LMArena Human Preference 140K

How to use LayoutLM models to compute image embeddings ?

  1. User

    How to use LayoutLM models to compute image embeddings ?

  2. Assistant

    Below you will find a practical “recipe” style walkthrough as well as a short conceptual explanation of what is really happening under the hood.
    Everything is done with 🤗 Transformers, so you only need Python, PyTorch (or TF/JAX) and PIL.


    0. TL;DR

    • Only LayoutLMv2, LayoutXLM and LayoutLMv3 contain a visual backbone capable of producing image / patch embeddings.
    • Use the corresponding Processor (LayoutLMv2Processor or LayoutLMv3Processor) so you don’t have to worry about image-normalisation or tokenisation.
    • The embeddings are returned in the model outputs:

    last_hidden_state               # sequence (cls + text tokens + visual patches)
    pooler_output / last_hidden_state[:,0,:]   # global (CLS) embedding
    

    1. Installation

    pip install transformers==4.39.3  pillow torchvision  # or torchaudio for JAX/TF
    

    2. Pick your model

    VariantVisual backboneTypical patch-gridProcessor class
    LayoutLMv22-stage ResNet-FPN → 14×14 patches (196)LayoutLMv2Processor
    LayoutLMv3Vision-Transformer 16×16 → 14×14 patches (196)LayoutLMv3Processor
    LayoutXLMSame as v2 (multilingual)LayoutXLMProcessor

    (Plain LayoutLM v1 has no visual branch → no image embeddings.)


    3. Quick code examples

    3.1 Pure image embedding (no text) — LayoutLMv3

    from transformers import LayoutLMv3Processor, LayoutLMv3Model
    from PIL import Image
    import torch
    
    # 1) Load model + processor
    model_name = "microsoft/layoutlmv3-base"
    processor  = LayoutLMv3Processor.from_pretrained(model_name)
    model      = LayoutLMv3Model.from_pretrained(model_name)
    
    # 2) Load image (RGB)
    img = Image.open("my_doc_page.jpg").convert("RGB")
    
    # 3) Pre-process ONLY the image
    inputs = processor(images=img, return_tensors="pt")  # no words / boxes
    
    # 4) Forward pass
    with torch.no_grad():
        outputs = model(**inputs)
    
    # 5) Get embeddings
    cls_embedding        = outputs.last_hidden_state[:, 0, :]        # (1, hidden)
    patch_embeddings     = outputs.last_hidden_state[:, 1:, :]       # (1, 196, hidden)
    
    # If you just need a global vector:
    doc_embedding = cls_embedding.squeeze(0)
    

    3.2 Image + OCR words/boxes — LayoutLMv2

    Useful when you later want a joint document embedding.

    from transformers import LayoutLMv2Processor, LayoutLMv2Model
    from PIL import Image
    import json, torch
    
    processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased")
    model     = LayoutLMv2Model.from_pretrained("microsoft/layoutlmv2-base-uncased")
    
    img  = Image.open("invoice.png").convert("RGB")
    
    # Example OCR output (use pytesseract, easyocr, Amazon Textract, …)
    words  = ["TOTAL", "5.99", "USD"]
    boxes  = [[42, 85,  95,105], [150,82,210,102], [215,85,270,105]]  # normalized 0-1000 or absolute pixel coords
    
    encoding = processor(img,
                         words,
                         boxes     = boxes,
                         max_length=512,
                         padding   ="max_length",
                         truncation=True,
                         return_tensors="pt")
    
    with torch.no_grad():
        out = model(**encoding)
    
    doc_embedding    = out.pooler_output.squeeze(0)  # or out.last_hidden_state[:,0,:]
    patch_embeddings = out.last_hidden_state[:, 1:1+196, :]
    

    4. What do these tensors represent?

    1. [CLS] token (index 0)
      • Mixes all modalities (text + layout + vision)
      • Good default “document-level” vector (dim = hidden_size, e.g. 768).

    2. Visual patches (next 196 positions for a 14×14 grid)
      • Each patch corresponds roughly to a 16×16 (v3) or 32×32 (v2) pixel square of the original 224×224 down-scaled image.
      • Shape (batch, N_patches, hidden). Useful for “visual-bag-of-patches”, attention pooling, cross-image similarity, etc.

    3. Text tokens (follow after visual tokens)
      • Same hidden dimension; carry fused textual, spatial and visual context.


    5. Saving / re-using the embedding

    torch.save(doc_embedding.cpu(), "invoice_emb.pt")   # PyTorch tensor
    # or
    import numpy as np, h5py
    np.save("invoice_emb.npy", doc_embedding.cpu().numpy())
    

    6. Frequently-asked questions

    Q. “Can I feed only an image to LayoutLMv2?”
    A. Yes. Supply empty words=[] and boxes=[], or use the feature-extractor directly:

    fe = LayoutLMv2Processor.from_pretrained(...).feature_extractor
    pixel_values = fe(img, return_tensors="pt")["pixel_values"]
    out = model(pixel_values=pixel_values, input_ids=None, bbox=None, attention_mask=None)
    

    Q. “Why 224×224? My original page is A4 size!”
    A. The model’s visual branch is fixed to 224×224 because it re-uses ImageNet pre-training.
    Information outside that resolution is lost, so many practitioners crop/slide windows or rely on the text tokens for details.

    Q. “I only want a pure image embedding, no text fusion.”
    A. LayoutLMv3’s CLS (with empty text) is already “pure”. If you need exactly the convolutional feature map, you can pull it from the model sub-modules:

    patch_emb = model.embeddings.patch_embeddings(pixel_values)   # (1, 196, hidden)
    

    7. Tips & gotchas

    • Normalise your bounding boxes to [0, 1000] × [0, 1000] if you use absolute pixel coords.
    • Padding/truncation rules matter: extra text tokens push the visual patch ids forward (CLS + 196 + …).
    • For retrieval tasks you often want the L2-normalised CLS vector:

    emb = torch.nn.functional.normalize(cls_embedding, p=2, dim=-1)
    

    Happy embedding! 🎉

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.