How to configure and run sglang on H200 openshift
- User
How to configure and run sglang on H200 openshift
- Assistant
Okay, let's break down how to configure and run a workload like
sglang(which often refers to libraries likevllmused for serving large language models efficiently) on H200 GPUs within an OpenShift cluster.Assumptions:
sglangrefers tovllm: I'll assume you mean using thevllmlibrary, as it's a popular high-performance serving framework often discussed in contexts similar to "sglang". If you mean a different specific tool namedsglang, the core OpenShift principles remain, but the application-specific steps (Dockerfile, entrypoint) will differ.- OpenShift Cluster: You have a running OpenShift cluster (version 4.x or later).
- H200 Nodes: Your cluster has nodes equipped with NVIDIA H200 GPUs.
- NVIDIA GPU Operator: The NVIDIA GPU Operator is installed and correctly configured in your cluster. This is essential for Kubernetes/OpenShift to recognize, manage, and allocate NVIDIA GPUs (including H200s). It installs drivers, CUDA libraries, DCGM exporter, etc.
- Access: You have
ocCLI access and appropriate permissions (e.g., cluster-admin for GPU Operator setup, project-admin for deploying your application).
Steps:
1. Prerequisites & Cluster Setup
- Verify H200 Nodes: Ensure the nodes with H200 GPUs are available and correctly recognized by OpenShift. The GPU Operator typically adds labels to nodes with GPUs.
# Check nodes and their labels oc get nodes # Look for labels like nvidia.com/gpu.product=H200, or nvidia.com/gpu=true oc adm node-label --list # To see labels applied by GPU Operator # Or specifically list nodes with GPUs oc get nodes -l nvidia.com/gpu=true - Install NVIDIA GPU Operator: If not already installed, follow the official OpenShift documentation to install it via OperatorHub or Helm. Ensure it's configured correctly for your cluster version and H200 hardware (usually auto-detects Hopper architecture).
- Create an OpenShift Project: Isolate your workload.
oc create project sglang-serving oc project sglang-serving - Container Registry: Ensure you have a container registry accessible from your OpenShift cluster (e.g., Quay.io, Harbor, or the internal OpenShift registry) where you can push your application image.
2. Containerize the vLLM Application
You need a container image that includes
vllmand your desired model.-
Dockerfile: Create a
Dockerfile.# Use an NVIDIA CUDA base image compatible with H200/Hopper (CUDA 12.x recommended) # Check NVIDIA NGC or CUDA Toolkit docs for the latest compatible image FROM nvidia/cuda:12.1.1-devel-ubuntu22.04 # Set environment variables to prevent interactive prompts during package installation ENV DEBIAN_FRONTEND=noninteractive # Install Python, pip, and build essentials RUN apt-get update && apt-get install -y --no-install-recommends \ python3.10 python3-pip python3.10-venv \ git wget build-essential && \ rm -rf /var/lib/apt/lists/* # Make python3 and pip point to the installed versions RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1 && \ update-alternatives --install /usr/bin/pip3 pip3 /usr/bin/pip3 1 # Install vLLM (check vllm documentation for specific install commands and recommended versions) # It's often best to install from source or a specific wheel for compatibility # Example using pip: RUN pip3 install --upgrade pip && \ pip3 install vllm # Or from source (example - adjust branch/commit as needed): # RUN git clone https://github.com/vllm-project/vllm.git /vllm # WORKDIR /vllm # RUN python3 setup.py install # WORKDIR / # Define model path (optional, can be mounted later) ENV MODEL_DIR=/models RUN mkdir -p $MODEL_DIR # Expose the API port EXPOSE 8000 # Default command to start the vLLM API server # Adjust --model path, tensor-parallelism, etc., as needed # Use --host 0.0.0.0 to listen on all interfaces within the container CMD ["python3", "-m", "vllm.entrypoints.api_server", \ "--model", "meta-llama/Llama-2-7b-chat-hf", \ "--tensor-parallel-size", "1", \ "--host", "0.0.0.0", \ "--port", "8000", \ "--disable-metadata-hook"] -
Build and Push Image:
- Option A (Using
ocbuild): If you have aBuildConfigset up. - Option B (Using
podmanordocker):# Login to your registry (replace with your registry details) podman login image-registry.openshift-image-registry.svc:5000 # Or your external registry # Build the image podman build -t image-registry.openshift-image-registry.svc:5000/sglang-serving/vllm-h200:latest . # Push the image podman push image-registry.openshift-image-registry.svc:5000/sglang-serving/vllm-h200:latest
Replace
image-registry.openshift-image-registry.svc:5000/sglang-serving/vllm-h200:latestwith your actual registry path. - Option A (Using
3. Define OpenShift Resources (Deployment & Service)
Create YAML files to define how
vllmruns in OpenShift.-
vllm-deployment.yaml: Defines the workload, including GPU resource requests.apiVersion: apps/v1 kind: Deployment metadata: name: vllm-deployment labels: app: vllm spec: replicas: 1 # Start with 1 replica, scale later if needed selector: matchLabels: app: vllm template: metadata: labels: app: vllm spec: # Optional: Use node affinity/selector to target H200 nodes if needed # affinity: # nodeAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # nodeSelectorTerms: # - matchExpressions: # - key: nvidia.com/gpu.product # Adjust key based on GPU Operator labels # operator: In # values: # - NVIDIA-H200 containers: - name: vllm # Use the image you pushed earlier image: image-registry.openshift-image-registry.svc:5000/sglang-serving/vllm-h200:latest ports: - containerPort: 8000 resources: limits: # Request 1 H200 GPU per pod nvidia.com/gpu: 1 # Specify CPU and Memory based on model needs and node capacity cpu: "4" # Example: Request 4 CPU cores memory: "16Gi" # Example: Request 16 GiB RAM requests: # Often good practice to match requests to limits for guaranteed resources nvidia.com/gpu: 1 cpu: "2" memory: "8Gi" # If mounting models from a Persistent Volume Claim (PVC): # volumeMounts: # - name: model-storage # mountPath: /models # readOnly: true # Optional: Define volumes if using PVCs # volumes: # - name: model-storage # persistentVolumeClaim: # claimName: h200-model-pvc # Replace with your PVC name # --- Important --- # If your model is large, ensure it fits in the H200's VRAM (e.g., 141 GB for H200) # vLLM manages GPU memory usage internally. You might tune parameters like: # --gpu-memory-utilization (default 0.9) # --max-model-len (e.g., 4096, 8192) depending on model and H200 capacity # For multi-GPU setups (Tensor Parallelism > 1), increase nvidia.com/gpu limit # and ensure the --tensor-parallel-size matches the number of GPUs requested. # Example: Request 2 GPUs and set TP=2 # resources: # limits: # nvidia.com/gpu: 2 # requests: # nvidia.com/gpu: 2 # ... inside the container spec ... # args: ["--tensor-parallel-size", "2"] # Override CMD if needed via args -
vllm-service.yaml: Exposes thevllmdeployment within the cluster.apiVersion: v1 kind: Service metadata: name: vllm-service spec: selector: app: vllm # Matches the labels on the Pods managed by the Deployment ports: - protocol: TCP port: 8000 # Port the Service listens on targetPort: 8000 # Port the container listens on type: ClusterIP # Exposes the service on a cluster-internal IP -
vllm-route.yaml(Optional): Exposes the service outside the cluster via an OpenShift Route.apiVersion: route.openshift.io/v1 kind: Route metadata: name: vllm-route spec: to: kind: Service name: vllm-service # Points to the Service created above port: targetPort: 8000 # The port defined in the Service # Optional: Configure TLS termination (e.g., Edge, Reencrypt, Passthrough) # tls: # termination: edge # insecureEdgeTerminationPolicy: Redirect # Or None, depending on needs # Hostname can be specified if needed, otherwise OpenShift assigns one # host: vllm.apps.yourcluster.yourdomain.com
4. Deploy to OpenShift
Apply the YAML manifests to your project:
oc apply -f vllm-deployment.yaml -f vllm-service.yaml # If external access is needed: oc apply -f vllm-route.yaml5. Monitor and Access
-
Check Deployment Status:
oc get deployments oc get pods oc logs <vllm-pod-name> # Check logs for errors during startup or inferenceWait until the pod is
Running. It might take time for the image to pull and the server to initialize. -
Check Service and Route:
oc get service vllm-service oc get route vllm-route # If created -
Access the API:
- Internal Access (within cluster): Use the service name:
http://vllm-service:8000 - External Access: Use the hostname provided by the Route:
http://<vllm-route-host>(e.g.,http://vllm-route-sglang-serving.apps.cluster.example.com)
- Internal Access (within cluster): Use the service name:
-
Test: Send a request using
curl.# Get the route hostname ROUTE_HOSTNAME=$(oc get route vllm-route -o jsonpath='{.spec.host}') # Example inference request (adapt payload based on model type) curl -X POST http://$ROUTE_HOSTNAME/generate \ -H "Content-Type: application/json" \ -d '{ "prompt": "An astronaut riding a horse on the moon.", "presence_penalty": 1.0, "max_tokens": 100, "temperature": 0.7 }' # Or check OpenAI compatible endpoint curl http://$ROUTE_HOSTNAME/v1/models
H200 Specific Considerations & Optimizations:
- Tensor Parallelism (TP): H200s have high memory bandwidth and NVLink interconnects. For very large models that require more VRAM than a single H200 provides or to increase throughput, use Tensor Parallelism.
- Modify the
Deployment: Increasenvidia.com/gpulimit (e.g., to2or4depending on node configuration) and adjust--tensor-parallel-sizein theargsorCMDof the container to match the number of GPUs requested. - Example
resourcesin Deployment:nvidia.com/gpu: 2 - Example
argsin Deployment:args: ["--model", "your-large-model", "--tensor-parallel-size", "2", ...]
- Modify the
- Model Weights: Ensure your model weights are accessible.
- Download on Startup: The Dockerfile example assumes downloading (
meta-llama/Llama-2-7b-chat-hf). This requires internet access and potentially Hugging Face login tokens (configure via secrets/env vars if needed). - Mount from PVC: For better management and faster startup, pre-download models to a Persistent Volume (PV) and create a Persistent Volume Claim (PVC). Mount this PVC into the pod using
volumesandvolumeMountsin the Deployment spec. This is generally the recommended approach for production.
- Download on Startup: The Dockerfile example assumes downloading (
- Resource Allocation:
- GPU Memory: H200 has 141 GB HBM3e. Monitor VRAM usage. Use vLLM parameters like
--gpu-memory-utilizationif you need to reserve less than the full GPU memory for the model, but typically you want to maximize utilization. - CPU/Memory: Allocate sufficient CPU and system RAM to the pod for data processing, communication, and the Python overhead.
- GPU Memory: H200 has 141 GB HBM3e. Monitor VRAM usage. Use vLLM parameters like
- Scheduling: Use
nodeSelectorornodeAffinityin the Deployment spec to guarantee pods land on nodes with H200s, especially if your cluster has heterogeneous hardware. Ensure the GPU Operator correctly labels the H200 nodes. - Monitoring: Utilize the DCGM Exporter metrics (exposed via Prometheus in OpenShift) to monitor GPU utilization, memory bandwidth, temperature, power draw, and NVLink utilization on the H200s.
Troubleshooting:
- Pod Pending: Often due to insufficient resources or node selectors not matching. Check
oc describe pod <pod-name>and ensure nodes have the required GPU labels and resources available. - ImagePullError: Check registry credentials and the image path. Ensure the internal OpenShift registry is configured correctly or external registry authentication is set up.
- Container Crash/Error: Examine
oc logs <pod-name>. Common issues include CUDA version mismatches, missing dependencies, incorrect model paths, or out-of-memory errors (either system RAM or GPU VRAM). - GPU Not Detected: Ensure the NVIDIA GPU Operator is running correctly and has configured the drivers/runtime. Check
nvidia-smiinside a pod if you temporarily run a privileged pod or use thedcgm-device-infocontainer.
This detailed guide should help you get
vllm(or a similar workload) running effectively on your H200 GPUs within OpenShift. Remember to adapt image names, model names, resource requests, and paths to your specific requirements.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.