how do I run docling (document to text service) in Kserve leveraging Kserve as much as…
- User
how do I run docling (document to text service) in Kserve leveraging Kserve as much as possible?
- Assistant
To run Docling, or any document-to-text service, in KServe, you need to deploy your model or service following KServe’s model inference standards. KServe (formerly KFServing) is a Kubernetes-based model serving platform built to simplify deploying and managing ML models.
Assuming Docling is a document-to-text pipeline or microservice (possibly a wrapper around Tesseract, OCR, PDF parsers, or NLP models), here’s how you can deploy it while leveraging KServe as much as possible.
✅ Assumptions:
- You have a Kubernetes cluster set up with KServe installed.
- You have Dockerfile or container image of Docling (or can build one).
- Your service can accept HTTP requests (ideally the KServe V2 inference protocol).
- You want autoscaling, GPU/CPU resource management, metrics, etc., via KServe.
🔧 Approach Overview:
- Containerize Docling
- Adapt the app to serve over HTTP (if not already)
- Conform to KServe v2 Inference Protocol (optional, but ideal)
- Deploy via InferenceService YAML
- Test & Monitor
1. 🐳 Containerize Docling
If Docling is not already in a Docker image:
Dockerfile Example:
FROM python:3.10-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . EXPOSE 8080 CMD ["python", "app.py"]You can include Flask/FastAPI and load the OCR model as needed.
2. 📡 Serve Over HTTP
Make sure your Docling service is accessible via HTTP — ideally FastAPI, Flask, or something lightweight:
Example (FastAPI):
from fastapi import FastAPI, UploadFile, File from docling import extract_text # Hypothetical module app = FastAPI() @app.post("/v1/models/docling:predict") async def predict(file: UploadFile = File(...)): content = await file.read() result = extract_text(content) # your logic return {"predictions": result}This mimics the
/v1/models/<model>:predictendpoint expected by KServe (v1 compatibility).
3. 🤖 Adapt to KServe V2 Protocol (optional but better)
If you want full KServe integration, follow KServe V2 inference protocol — especially useful if you're integrating with other model pipelines.
Else, just match minimal v1 pattern like above.
4. 🚀 Deploy with InferenceService
KServe uses an InferenceService Custom Resource to deploy models.
Here’s an example YAML:
apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: docling-service spec: predictor: containers: - name: docling-container image: your-dockerhub-username/docling:v1 ports: - containerPort: 8080 command: ["python"] args: ["app.py"] resources: requests: cpu: "1" memory: "1Gi"🔍 Replace
your-dockerhub-username/docling:v1with your actual image registry path.Apply the YAML:
kubectl apply -f docling-inferenceservice.yaml
5. 🔬 Test the Endpoint
Once deployed, you can test the endpoint using:
kubectl get inferenceservice docling-service # Port-forward for debugging kubectl port-forward svc/docling-service-predictor-default 8080:80Then:
curl -X POST http://localhost:8080/v1/models/docling:predict \ -F "file=@example.pdf"
✅ Leveraging KServe Features
By using KServe you get:
- Auto-scaling (including to zero)
- Resource management (CPU/GPU, memory)
- Model metrics + observability
- Canary rollouts
- Multi-model support
- Python, TF, PyTorch, and custom runtimes
🌟 Bonus: Advanced Features You Can Add
Feature Benefit How To Use Explainers Add model explainability Add explainer:in InferenceServiceTransformers Pre/Post processors (e.g., image preprocessing) Add transformer:Batcher Accumulate requests to batch-process Via InferenceService spec GPU Support Leverage GPU for OCR/NLP Add resources.limits.nvidia.com/gpu
🧠 Summary
You can run Docling in KServe by:
- Creating a containerized version
- Serving it via HTTP (FastAPI recommended)
- Deploying it with an InferenceService
- Testing the endpoint
- Optionally conforming to KServe V2 inference protocol
This leverages KServe’s autoscaling, versioning, and Kubernetes-native model management.
If you need help creating the container or adapting to KServe protocol, feel free to drop in code snippets or more details of Docling!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.