How to use LayoutLM models to compute image embeddings ?
- User
How to use LayoutLM models to compute image embeddings ?
- Assistant
Below you will find a practical “recipe” style walkthrough as well as a short conceptual explanation of what is really happening under the hood.
Everything is done with 🤗 Transformers, so you only need Python, PyTorch (or TF/JAX) and PIL.
0. TL;DR
• Only LayoutLMv2, LayoutXLM and LayoutLMv3 contain a visual backbone capable of producing image / patch embeddings.
• Use the corresponding Processor (LayoutLMv2ProcessororLayoutLMv3Processor) so you don’t have to worry about image-normalisation or tokenisation.
• The embeddings are returned in the model outputs:last_hidden_state # sequence (cls + text tokens + visual patches) pooler_output / last_hidden_state[:,0,:] # global (CLS) embedding
1. Installation
pip install transformers==4.39.3 pillow torchvision # or torchaudio for JAX/TF
2. Pick your model
Variant Visual backbone Typical patch-grid Processor class LayoutLMv2 2-stage ResNet-FPN → 14×14 patches (196) LayoutLMv2ProcessorLayoutLMv3 Vision-Transformer 16×16 → 14×14 patches (196) LayoutLMv3ProcessorLayoutXLM Same as v2 (multilingual) LayoutXLMProcessor(Plain LayoutLM v1 has no visual branch → no image embeddings.)
3. Quick code examples
3.1 Pure image embedding (no text) — LayoutLMv3
from transformers import LayoutLMv3Processor, LayoutLMv3Model from PIL import Image import torch # 1) Load model + processor model_name = "microsoft/layoutlmv3-base" processor = LayoutLMv3Processor.from_pretrained(model_name) model = LayoutLMv3Model.from_pretrained(model_name) # 2) Load image (RGB) img = Image.open("my_doc_page.jpg").convert("RGB") # 3) Pre-process ONLY the image inputs = processor(images=img, return_tensors="pt") # no words / boxes # 4) Forward pass with torch.no_grad(): outputs = model(**inputs) # 5) Get embeddings cls_embedding = outputs.last_hidden_state[:, 0, :] # (1, hidden) patch_embeddings = outputs.last_hidden_state[:, 1:, :] # (1, 196, hidden) # If you just need a global vector: doc_embedding = cls_embedding.squeeze(0)3.2 Image + OCR words/boxes — LayoutLMv2
Useful when you later want a joint document embedding.
from transformers import LayoutLMv2Processor, LayoutLMv2Model from PIL import Image import json, torch processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased") model = LayoutLMv2Model.from_pretrained("microsoft/layoutlmv2-base-uncased") img = Image.open("invoice.png").convert("RGB") # Example OCR output (use pytesseract, easyocr, Amazon Textract, …) words = ["TOTAL", "5.99", "USD"] boxes = [[42, 85, 95,105], [150,82,210,102], [215,85,270,105]] # normalized 0-1000 or absolute pixel coords encoding = processor(img, words, boxes = boxes, max_length=512, padding ="max_length", truncation=True, return_tensors="pt") with torch.no_grad(): out = model(**encoding) doc_embedding = out.pooler_output.squeeze(0) # or out.last_hidden_state[:,0,:] patch_embeddings = out.last_hidden_state[:, 1:1+196, :]
4. What do these tensors represent?
-
[CLS]token (index 0)
• Mixes all modalities (text + layout + vision)
• Good default “document-level” vector (dim = hidden_size, e.g. 768). -
Visual patches (next 196 positions for a 14×14 grid)
• Each patch corresponds roughly to a 16×16 (v3) or 32×32 (v2) pixel square of the original 224×224 down-scaled image.
• Shape (batch, N_patches, hidden). Useful for “visual-bag-of-patches”, attention pooling, cross-image similarity, etc. -
Text tokens (follow after visual tokens)
• Same hidden dimension; carry fused textual, spatial and visual context.
5. Saving / re-using the embedding
torch.save(doc_embedding.cpu(), "invoice_emb.pt") # PyTorch tensor # or import numpy as np, h5py np.save("invoice_emb.npy", doc_embedding.cpu().numpy())
6. Frequently-asked questions
Q. “Can I feed only an image to LayoutLMv2?”
A. Yes. Supply emptywords=[]andboxes=[], or use the feature-extractor directly:fe = LayoutLMv2Processor.from_pretrained(...).feature_extractor pixel_values = fe(img, return_tensors="pt")["pixel_values"] out = model(pixel_values=pixel_values, input_ids=None, bbox=None, attention_mask=None)Q. “Why 224×224? My original page is A4 size!”
A. The model’s visual branch is fixed to 224×224 because it re-uses ImageNet pre-training.
Information outside that resolution is lost, so many practitioners crop/slide windows or rely on the text tokens for details.Q. “I only want a pure image embedding, no text fusion.”
A. LayoutLMv3’s CLS (with empty text) is already “pure”. If you need exactly the convolutional feature map, you can pull it from the model sub-modules:patch_emb = model.embeddings.patch_embeddings(pixel_values) # (1, 196, hidden)
7. Tips & gotchas
• Normalise your bounding boxes to
[0, 1000] × [0, 1000]if you use absolute pixel coords.
• Padding/truncation rules matter: extra text tokens push the visual patch ids forward (CLS + 196 + …).
• For retrieval tasks you often want the L2-normalised CLS vector:emb = torch.nn.functional.normalize(cls_embedding, p=2, dim=-1)
Happy embedding! 🎉
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.