Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

how, from technical point of view, will all of this information be used by the model…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-235b-a22b’s answer.
  1. User

    how, from technical point of view, will all of this information be used by the model when it encounters a new template. the data will not be positioined in the same place. the number of lines will not be the same. how will it be able to tell the buyer address if it's going to be in a different place on each template?

  2. Assistant

    Below is a “nuts-and-bolts” explanation of HOW a modern document-understanding model (e.g. LayoutLM-v2/-v3, DocFormer, Donut, FormNet, etc.) can still spot “Buyer address”, “Ship-To”, and variable-length order lines even when every template looks different.

    ────────────────────────────────────────

    1. What features actually reach the model? ──────────────────────────────────────── After OCR / layout parsing each token is represented as a rich vector that combines:

    • Text token embedding
    – The usual sub-word ID from a WordPiece/BPE vocabulary.

    • 2-D position (geometry)
    – xmin, ymin, xmax, ymax for the token’s bounding box in pixel or normalized page space.
    – From this, the library creates either absolute position embeddings or relative 2-D biases (Δx, Δy, Δw, Δh) between every pair of tokens.

    • Page / segment ID
    – Lets the model know which page if the doc is multi-page.

    • Visual patch features (v2+ models)
    – A small CNN is run on the image region around the token; the resulting vector is concatenated.

    These pieces are concatenated / summed to get the final vector Eᵢ that is fed to the Transformer.

    ──────────────────────────────────────── 2. Why does this generalise to new layouts? ──────────────────────────────────────── A) Layout awareness is relative, not absolute
    – Self-attention sees pairwise geometry: “token A is 200 px below & 30 px right of token B”.
    – So “Buyer address” can sit top-left on Template-1 and bottom-right on Template-2; the model mainly looks for the LOCAL pattern “lines that follow a cue like ‘Bill To’ and resemble an address”.

    B) Content cues > pixel coordinates
    – Transformers can attend to lexical anchors (“Bill To”, “Ship To”, “Buyer”, “PO#”, currency symbols, etc.).
    – During fine-tuning the gradient tells the model: “when you see token sequence shaped like an address right after anchor X, label it BUYER_ADDRESS”.
    – Even without an explicit anchor, address-like patterns (street name, ZIP, country) are strongly informative.

    C) Order-line variability is naturally handled
    – The model is token or span classifier, so it makes a label for every token no matter how long the table is.
    – A document with 3 rows simply produces more tokens labelled I-LINE_ITEM_* than a doc with 20 rows.

    ──────────────────────────────────────── 3. Training objective that ties it together ──────────────────────────────────────── Token-level cross-entropy:
    Given sequence 𝒯 = [t₁, … tₙ] with labels 𝒴 = [y₁, … yₙ] (BIO scheme), minimize

    L = − Σᵢ log p( yᵢ | 𝒯 )
    

    The model therefore learns to produce the same label for a “123 Main St” span regardless of where it shows up, as long as the surrounding visual-textual pattern is similar to examples in your 70 annotated docs.

    ──────────────────────────────────────── 4. Inference on a brand-new template ──────────────────────────────────────── Step-by-step:

    1. OCR + LayoutParser ⇒ tokens with boxes
    2. Tokens + boxes ⇒ Embed (text + 2-D + visual)
    3. Transformer ⇒ contextual representations h₁…hₙ
    4. Classification head ⇒ probabilities over {B-BUYER_ADDR, I-…, O, …}.
    5. Post-process
      • Merge consecutive B/I to full spans.
      • For order lines, cluster by Y-coordinate ⇒ build rows, then parse columns (description / qty / price).
    6. XML builder writes the final structured output.

    Notice that step 4 gives a label for every token, independently of the absolute page coordinates. Step 5 contains light heuristics (row clustering) but MOST of the heavy lifting—figuring out which tokens belong to which field—already happened in the model.

    ──────────────────────────────────────── 5. Continuous learning for unseen forms ──────────────────────────────────────── When the UI user corrects mistakes:

    • Store (token, bbox, corrected_label) triples.
    • Append to the fine-tuning dataset.
    • Re-fine-tune ↻.

    Because the 2-D embeddings are learned, the model gradually absorbs brand-new spatial configurations encountered in the corrected samples.

    ──────────────────────────────────────── 6. Summary “mental model” ──────────────────────────────────────── Think of the network as:

    “I read what the token says and who is near me (in 2-D space).
    Over dozens of POs I learned the universal local pattern   anchor → multi-line address tokens
      header → tabular rows for items
    So wherever that pattern reappears—even lower-left corner on a brand-new vendor form—I will tag it correctly.”

    That is why you do NOT hard-code template rules; you equip the network with content + layout features and feed it diverse annotated examples.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.