Skip to content
The Chat Archive
chatgpt-4o-latest-20250326· OpenAI LMArena Human Preference 140K

how do I run docling (document to text service) in Kserve leveraging Kserve as much as…

  1. User

    how do I run docling (document to text service) in Kserve leveraging Kserve as much as possible?

  2. Assistant

    To run Docling, or any document-to-text service, in KServe, you need to deploy your model or service following KServe’s model inference standards. KServe (formerly KFServing) is a Kubernetes-based model serving platform built to simplify deploying and managing ML models.

    Assuming Docling is a document-to-text pipeline or microservice (possibly a wrapper around Tesseract, OCR, PDF parsers, or NLP models), here’s how you can deploy it while leveraging KServe as much as possible.


    ✅ Assumptions:

    • You have a Kubernetes cluster set up with KServe installed.
    • You have Dockerfile or container image of Docling (or can build one).
    • Your service can accept HTTP requests (ideally the KServe V2 inference protocol).
    • You want autoscaling, GPU/CPU resource management, metrics, etc., via KServe.

    🔧 Approach Overview:

    1. Containerize Docling
    2. Adapt the app to serve over HTTP (if not already)
    3. Conform to KServe v2 Inference Protocol (optional, but ideal)
    4. Deploy via InferenceService YAML
    5. Test & Monitor

    1. 🐳 Containerize Docling

    If Docling is not already in a Docker image:

    Dockerfile Example:

    FROM python:3.10-slim
    
    WORKDIR /app
    
    COPY requirements.txt .
    RUN pip install -r requirements.txt
    
    COPY . .
    
    EXPOSE 8080
    CMD ["python", "app.py"]
    

    You can include Flask/FastAPI and load the OCR model as needed.


    2. 📡 Serve Over HTTP

    Make sure your Docling service is accessible via HTTP — ideally FastAPI, Flask, or something lightweight:

    Example (FastAPI):

    from fastapi import FastAPI, UploadFile, File
    from docling import extract_text  # Hypothetical module
    
    app = FastAPI()
    
    @app.post("/v1/models/docling:predict")
    async def predict(file: UploadFile = File(...)):
        content = await file.read()
        result = extract_text(content)  # your logic
        return {"predictions": result}
    

    This mimics the /v1/models/<model>:predict endpoint expected by KServe (v1 compatibility).


    3. 🤖 Adapt to KServe V2 Protocol (optional but better)

    If you want full KServe integration, follow KServe V2 inference protocol — especially useful if you're integrating with other model pipelines.

    Else, just match minimal v1 pattern like above.


    4. 🚀 Deploy with InferenceService

    KServe uses an InferenceService Custom Resource to deploy models.

    Here’s an example YAML:

    apiVersion: serving.kserve.io/v1beta1
    kind: InferenceService
    metadata:
      name: docling-service
    spec:
      predictor:
        containers:
          - name: docling-container
            image: your-dockerhub-username/docling:v1
            ports:
              - containerPort: 8080
            command: ["python"]
            args: ["app.py"]
            resources:
              requests:
                cpu: "1"
                memory: "1Gi"
    

    🔍 Replace your-dockerhub-username/docling:v1 with your actual image registry path.

    Apply the YAML:

    kubectl apply -f docling-inferenceservice.yaml
    

    5. 🔬 Test the Endpoint

    Once deployed, you can test the endpoint using:

    kubectl get inferenceservice docling-service
    
    # Port-forward for debugging
    kubectl port-forward svc/docling-service-predictor-default 8080:80
    

    Then:

    curl -X POST http://localhost:8080/v1/models/docling:predict \
      -F "file=@example.pdf"
    

    ✅ Leveraging KServe Features

    By using KServe you get:

    • Auto-scaling (including to zero)
    • Resource management (CPU/GPU, memory)
    • Model metrics + observability
    • Canary rollouts
    • Multi-model support
    • Python, TF, PyTorch, and custom runtimes

    🌟 Bonus: Advanced Features You Can Add

    FeatureBenefitHow To Use
    ExplainersAdd model explainabilityAdd explainer: in InferenceService
    TransformersPre/Post processors (e.g., image preprocessing)Add transformer:
    BatcherAccumulate requests to batch-processVia InferenceService spec
    GPU SupportLeverage GPU for OCR/NLPAdd resources.limits.nvidia.com/gpu

    🧠 Summary

    You can run Docling in KServe by:

    1. Creating a containerized version
    2. Serving it via HTTP (FastAPI recommended)
    3. Deploying it with an InferenceService
    4. Testing the endpoint
    5. Optionally conforming to KServe V2 inference protocol

    This leverages KServe’s autoscaling, versioning, and Kubernetes-native model management.


    If you need help creating the container or adapting to KServe protocol, feel free to drop in code snippets or more details of Docling!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.