Skip to content
TheChatArchiveThe Chat Archive
chatgpt-4o-latest-20250326· OpenAIHugging Face LMArena Human Preference 140K

When I use multimodal embeddings, what skill should I use: Document Extraction or…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219’s answer.
  1. User

    When I use multimodal embeddings, what skill should I use: Document Extraction or Document Layout?

  2. Assistant

    Great question — when using multimodal embeddings in an Azure AI Search pipeline for PDF documents with both text and images, the decision between using the Document Extraction skill vs. Document Layout skill significantly affects the quality of the context passed into your embeddings.

    Here’s the direct recommendation ➝


    ✅ Use the Document Layout Skill (for multimodal embeddings)

    Why?

    Multimodal embeddings (e.g., with CLIP or Florence) benefit from preserving the visual structure of the original document — including:

    • Image positions and associated text
    • Layouts like tables, headers, footnotes
    • Multi-column formats
    • UI screenshots or flow diagrams with captions

    The Document Layout Skill provides structured, page-aware, layout-preserving chunks, whereas Document Extraction provides only a linear “plain text” stream.


    🔍 Skill Differences in Detail

    FeatureDocument Extraction SkillDocument Layout Skill
    🧠 PurposeExtracts raw text from documentsExtracts structured layout-aware content
    💡 OutputPlain JSON with text contentHierarchical representation with regions, text blocks, images
    🖼️ Visual Context❌ No✅ Yes (image/text positioning preserved)
    📷 Image Association❌ Hard to retain image context✅ Maintains image-to-text proximity
    🧱 Best ForSimple, text-only PDFsComplex, visual-rich PDFs (like your handbook)
    🤝 Multimodal Embedding SupportNot idealRecommended

    📘 Example: Your Use Case — Technical Handbook with Explanatory Images

    Your PDF includes:

    • Step-by-step instructions alongside screenshots
    • Visual elements like callouts, arrows, interface highlights
    • Possibly multi-column formatting or diagrams

    The Document Layout Skill will:

    • Extract both text and images, along with their layout coordinates
    • Allow you to chunk and embed data in a semantically meaningful, multimodal way (good for Azure Search + RAG)
    • Retain image-to-caption proximity — huge benefit for multimodal embeddings like CLIP or Florence

    Tip: Use Skillsets in Azure Cognitive Search

    In your enrichment pipeline (via Azure Search skillsets):

    • Add the Document Layout skill
    • Use Semantic Chunking or a Custom Skill to prepare content for embedding if needed
    • Feed combined text-image-layout segments into Multimodal Embedding skill

    ✅ Final Answer

    Use the Document Layout skill when generating content for multimodal embeddings of PDF documents.
    Because it retains visual structure, image associations, and positional context — essential for a high-quality RAG pipeline that understands and retrieves semantically and visually rich content.


    Let me know if you want YAML/JSON templates for setting up that pipeline in Azure Search!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.