Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Compose a multi-tenant inference sandbox isolating secrets, monitoring per-tenant SLAs,…

  1. User

    Compose a multi-tenant inference sandbox isolating secrets, monitoring per-tenant SLAs, and throttling.

  2. GLM Assistant

    The user is asking for a "multi-tenant inference sandbox." The key requirements are:

    1. Isolating secrets: Each tenant's credentials/API keys must be kept separate and secure.
    2. Monitoring per-tenant SLAs: We need to track metrics (latency, throughput, errors) specific to each tenant.
    3. Throttling: Limiting resource usage (requests per second, concurrent requests, compute) per tenant to prevent abuse or "noisy neighbor" issues.

    This sounds like a system design and implementation question. I need to provide a conceptual architecture and likely some code snippets to demonstrate how this would work.

    Target Audience: Likely a software engineer or architect looking for a pattern or a starting point for building such a system.

    High-Level Architecture:

    • API Gateway / Load Balancer: The entry point. Responsible for authentication, routing, and initial throttling.
    • Sandbox Controller/Orchestrator: The core logic that manages the lifecycle of inference requests, enforces policies, and isolates contexts.
    • Inference Engine: The actual ML model runtime (e.g., vLLM, TensorRT-LLM, PyTorch).
    • Secrets Manager: External secure storage (e.g., HashiCorp Vault, AWS Secrets Manager).
    • Metrics/Observability Stack: Prometheus, Grafana, etc.

    Key Components & Implementation Details:

    1. Secrets Isolation:

      • Problem: We don't want Tenant A to access Tenant B's API keys or prompt data.
      • Solution: Never store secrets in the code. Fetch them just-in-time (JIT) or mount them via volume/environment variables in isolated containers/pods. For an inference API, often the secrets are the API keys to call the inference, or perhaps the inference engine calls an external tool (e.g., Google Search) requiring a key.
      • Approach: Use a context object for each request that holds the tenant ID. Use that ID to fetch specific secrets from a secure vault at runtime.
    2. Monitoring (SLAs):

      • Problem: We need to know if Tenant A is getting 99.9% uptime while Tenant B is getting 50%.
      • Solution: Label all metrics with tenant_id.
      • Tools: Prometheus is standard here. We need middleware in the application to record latency, error rates, and token counts per tenant.
    3. Throttling:

      • Problem: Tenant A sends 1M requests and starves Tenant B.
      • Solution: Token Bucket or Leaky Bucket algorithms. Redis is often used for distributed rate limiting.
      • Granularity: RPM (Requests Per Minute), TPM (Tokens Per Minute), Concurrent Requests.

    Structuring the Code/Response:

    I will create a Python-based pseudo-framework using FastAPI as the web layer, as it's standard for modern ML serving. I'll simulate the inference engine part.

    • Class: TenantContext: Holds config (rate limits, secrets reference).
    • Class: SecretsManager: Mock interaction with a vault.
    • Class: RateLimiter: Redis-backed (mocked) token bucket.
    • Class: MetricsMiddleware: Prometheus metrics generation.
    • Class: InferenceSandbox: The core execution wrapper.

    Drafting the Code Structure:

    • Imports: FastAPI, Prometheus Client, Redis (or mock).
    • Setup: Define Tenant configs.
    • Middleware: Logic to extract Tenant ID from headers, verify API Key, and start metrics timer.
    • Dependency Injection: Inject the TenantContext.
    • Rate Limiting: Check Redis before processing. If fail, raise 429.
    • Execution:
      1. Fetch Secret (if needed for the model, though usually inference models are internal, maybe the tenant defines a specific RAG source? Let's assume the "secret" is an external API key the model might use during inference, like a tool).
      2. Run Inference (mock time.sleep).
      3. Record Metrics.
    • Cleanup: Ensure no data leaks between requests (Python globals are a risk, need to be careful).

    Refining the "Sandbox" Aspect: A true "sandbox" implies code execution isolation (like Docker/gVisor). However, in the context of "Inference," it usually means logical isolation (namespaces) and resource quotas (GPU memory allocation). I will interpret "Sandbox" as the logical wrapper around the inference call that enforces these constraints. If I were to do process isolation, that's a much longer answer (Kubernetes operators). I will stick to the application-level orchestration which is usually what's requested in a prompt like this, but mention Kubernetes/Docker for the physical isolation.

    Step-by-Step Implementation Plan:

    1. Define the Tenant Model: ID, Rate Limits, Secrets Path.
    2. Secret Manager: A class that simulates fetching secrets based on Tenant ID, ensuring isolation.
    3. Rate Limiter: A Redis-based sliding window or token bucket logic.
    4. Metrics: A custom Prometheus exporter wrapper that tags metrics by tenant.
    5. The Application: FastAPI app connecting these pieces.

    Self-Correction/Refinement during drafting:

    • Initial thought: Should I use actual Docker SDK to spin up containers?
    • Correction: No, that's too heavy for a code snippet. Better to build a "Logical Sandbox" class that represents the control plane.
    • SLA Monitoring: Simply logging isn't enough. I need to expose a /metrics endpoint.
    • Secrets: I'll assume the "inference" needs to call an external API using a tenant-specific key (e.g., a RAG system connecting to the tenant's private database).

    Code Sketch (Mental):

    app = FastAPI()
    
    # Global store for tenants
    tenants = {
        "tenant_a": TenantConfig(rate_limit=10, secret_id="vault/secret/a")
    }
    
    @app.middleware
    async def tenant_resolver(request, call_next):
        tenant_id = request.headers['X-Tenant-ID']
        # Validate, Rate Limit, Metrics Start
        ...
        response = await call_next(request)
        # Metrics End
        return response
    

    Let's make it more robust. I'll create a SandboxOrchestrator class.

    Final Plan:

    1. Architecture Overview: Explain the separation of concerns (Gateway vs. App vs. Infra).
    2. Implementation: A Python implementation using FastAPI, Redis (via mock), and Prometheus.
      • Isolation: Context variables (contextvars) to ensure request data doesn't leak across threads in async Python.
      • Secrets: A Vault abstraction.
      • SLA: Latency histograms per tenant.
      • Throttling: Dependency injection for rate limiting.

    Let's write the code. I need to ensure contextvars is used to prevent leaking sensitive data in async concurrent environments.

    Key considerations for "Sandbox" in Multi-tenant Inference:

    • Memory: You can't easily isolate memory in pure Python without separate processes. I will note that GPU memory management usually requires a specialized inference server (like vLLM with gpu_memory_utilization per tenant) or distinct pods. I will implement request throttling to control throughput/queue depth.

    Constructing the response:

    1. System Design Diagram (Text-based).
    2. Python Code (FastAPI + Prometheus + Redis logic).
    3. Explanation of the "Sandbox" controls.

    Let's refine the Python code to be production-grade logic (pseudo-code for external dependencies like Redis/Vault to keep it runnable or readable).

    • Dependency: fastapi, prometheus-client.
    • Rate Limiting: Use a TokenBucket class.
    • Secrets: get_secret(tenant_id).
    • SLA: Histogram for latency.

    Let's assemble.

    Self-Correction during code generation: Ensure I handle the case where a tenant is not found. Ensure the metrics are specific (e.g., inference_latency_seconds{tenant="A"}).

    The prompt asks to "Compose". This implies a holistic solution, likely code-heavy.

    Plan:

    1. Architecture Description: Explain the layers (Ingress, Control Plane, Runtime).
    2. Python Implementation: A FastAPI service acting as the Sandbox Controller.
      • TenantManager: Config/Secrets.
      • RateLimiter: Throttling logic.
      • Metrics: SLA tracking.
      • InferenceRunner: The actual "work".
    3. Security Note: Explain how contextvars and process isolation work.

    Let's go. This design implements a Logical Sandbox architecture. In a multi-tenant AI inference environment, complete physical isolation (one container per request) is often too slow and expensive. Instead, we use a logical sandbox that enforces strict boundaries at the application layer.

    Architecture Overview

    1. Authentication & Context Injection: Validates the tenant and injects a TenantContext object into the request scope.
    2. Governance Layer (The "Sandbox"):
      • Secrets: Fetches tenant-specific secrets (e.g., Vector DB keys, External APIs) just-in-time from a Vault. These are never stored in global variables.
      • Throttling: Uses a Token Bucket algorithm (Redis-backed) to enforce RPM (Requests Per Minute) and TPM (Tokens Per Minute).
    3. Observability: Wraps the inference call to emit metrics (Latency, Throughput, Errors) tagged by tenant_id.
    4. Execution: Runs the model inference within a timeout wrapper to prevent hogging.

    Implementation (Python FastAPI)

    Here is a complete, runnable simulation of this architecture.

    import asyncio
    import time
    import uuid
    import json
    from dataclasses import dataclass, field
    from typing import Optional, Dict
    from fastapi import FastAPI, Header, HTTPException, Request
    from contextvars import ContextVar
    from prometheus_client import Counter, Histogram, generate_latest
    from redis import Redis
    # Mocking libraries for demonstration purposes
    # In production: use prometheus_client, redis, hvac (for vault)
    
    # --- 1. Context & Security ---
    
    # ContextVar ensures tenant data is isolated to the specific async request thread
    current_tenant_ctx: ContextVar[str] = ContextVar('tenant_id', default=None)
    
    @dataclass
    class TenantConfig:
        id: str
        tier: str  # e.g., 'free', 'pro', 'enterprise'
        rate_limit_rpm: int
        allowed_models: list
        # Reference to secret path, NOT the secret itself
        secret_path: str 
    
    # Mock Database of Tenants
    TENANT_DB = {
        "tenant_alpha": TenantConfig(
            id="tenant_alpha", tier="enterprise", rate_limit_rpm=100, 
            allowed_models=["gpt-4", "bert-base"], secret_path="secrets/alpha"
        ),
        "tenant_beta": TenantConfig(
            id="tenant_beta", tier="free", rate_limit_rpm=10, 
            allowed_models=["bert-base"], secret_path="secrets/beta"
        )
    }
    
    class SecretManager:
        """
        Simulates HashiCorp Vault or AWS Secrets Manager.
        Secrets are fetched dynamically and discarded after use.
        """
        async def get_inference_secret(self, path: str) -> str:
            # In reality: vault_client.read(path)
            print(f"--> [SECURITY] Fetching secret from path: {path}")
            await asyncio.sleep(0.01) # Simulate network IO
            return f"API_KEY_FOR_{path}"
    
    # --- 2. SLA Monitoring (Prometheus Mock) ---
    
    # Metrics must be initialized once
    INFERENCE_COUNT = Counter('inference_requests_total', 'Total Inference Requests', ['tenant_id', 'model', 'status'])
    INFERENCE_LATENCY = Histogram('inference_latency_seconds', 'Inference Latency', ['tenant_id', 'model'])
    
    # --- 3. Throttling (Token Bucket via Redis) ---
    
    class RateLimiter:
        def __init__(self, redis_client: Redis):
            self.redis = redis_client
    
        async def is_allowed(self, tenant_id: str, limit: int) -> bool:
            """
            Simple sliding window log algorithm.
            Key: tenant_id:requests
            Value: Timestamps of requests
            """
            now = time.time()
            window_start = now - 60
            key = f"rate:{tenant_id}"
    
            # Lua script is preferred in production for atomicity
            # Here we simulate the logic in Python for readability
            
            # 1. Remove old entries
            # self.redis.zremrangebyscore(key, 0, window_start) 
            
            # 2. Count current entries
            # current_count = self.redis.zcard(key)
            
            # MOCK IMPLEMENTATION:
            current_count = 0 # Assume 0 for demo freshness
            
            if current_count < limit:
                # self.redis.zadd(key, {str(now): now})
                # self.redis.expire(key, 60)
                return True
            return False
    
    # --- 4. The Sandbox Orchestestrator ---
    
    class InferenceSandbox:
        def __init__(self, secret_manager: SecretManager, rate_limiter: RateLimiter):
            self.secret_manager = secret_manager
            self.rate_limiter = rate_limiter
    
        async def run_inference(self, tenant: TenantConfig, model_id: str, prompt: str):
            start_time = time.time()
            
            # A. Isolation Check: Model Access
            if model_id not in tenant.allowed_models:
                INFERENCE_COUNT.labels(tenant_id=tenant.id, model=model_id, status="forbidden").inc()
                raise HTTPException(status_code=403, detail="Model not available for this tier")
    
            # B. Throttling
            if not await self.rate_limiter.is_allowed(tenant.id, tenant.rate_limit_rpm):
                INFERENCE_COUNT.labels(tenant_id=tenant.id, model=model_id, status="throttled").inc()
                raise HTTPException(status_code=429, detail="Rate limit exceeded")
    
            # C. Secret Injection (Just-In-Time)
            # We fetch the secret only when we are sure the request is valid
            inference_api_key = await self.secret_manager.get_inference_secret(tenant.secret_path)
    
            try:
                # D. Execution with SLA Monitoring
                with INFERENCE_LATENCY.labels(tenant_id=tenant.id, model=model_id).time():
                    # Simulate Inference Call using the injected secret
                    # result = await external_model_client.call(model_id, prompt, key=inference_api_key)
                    print(f"    [EXECUTING] Tenant: {tenant.id} | Model: {model_id} | Key: {inference_api_key[:20]}...")
                    await asyncio.sleep(0.5) # Simulate model inference time
                
                INFERENCE_COUNT.labels(tenant_id=tenant.id, model=model_id, status="success").inc()
                return {"result": "Generated text based on prompt..."}
    
            except Exception as e:
                INFERENCE_COUNT.labels(tenant_id=tenant.id, model=model_id, status="error").inc()
                raise HTTPException(status_code=500, detail=str(e))
            finally:
                # E. Secret Cleanup (Logic)
                # In Python, variables are garbage collected. In a strict sandbox,
                # you might explicitly wipe the memory here, but usually scope exit is sufficient.
                del inference_api_key
    
    # --- 5. FastAPI Application ---
    
    app = FastAPI()
    # mock_redis = Redis(host='localhost', port=6379, db=0)
    mock_redis = None # Using mock logic in RateLimiter for this example
    
    secret_mgr = SecretManager()
    limiter = RateLimiter(mock_redis)
    sandbox = InferenceSandbox(secret_mgr, limiter)
    
    @app.post("/v1/inference/{model_id}")
    async def inference_endpoint(
        request: Request, 
        model_id: str, 
        x_tenant_id: str = Header(...)
    ):
        # 1. Resolve Tenant
        tenant = TENANT_DB.get(x_tenant_id)
        if not tenant:
            raise HTTPException(status_code=401, detail="Unknown Tenant")
    
        # 2. Set Context Var (Useful for logging middlewares)
        current_tenant_ctx.set(tenant.id)
        
        # 3. Parse Payload
        body = await request.json()
        prompt = body.get("prompt")
    
        # 4. Enter Sandbox
        result = await sandbox.run_inference(tenant, model_id, prompt)
        
        return result
    
    @app.get("/metrics")
    async def metrics():
        # Prometheus scrape endpoint
        return generate_latest()
    
    

    Key Isolation Mechanisms Explained

    1. Secret Isolation (The "Vault" Pattern)

      • Problem: If you store API keys in environment variables (os.environ), every tenant potentially has access to every key.
      • Sandbox Solution: The SecretManager class fetches secrets dynamically based on the tenant.secret_path. The key exists only in the local variable inference_api_key during the execution of run_inference. Once that function returns, the variable is destroyed. If Tenant A sends a prompt, they physically cannot induce the code to fetch Tenant B's key because the code path depends on the authenticated x_tenant_id.
    2. SLA Monitoring (Metrics)

      • Problem: Aggregating metrics across all tenants hides individual failures. If the average latency is 100ms, but Tenant A is 1ms and Tenant B is 10s, you won't know Tenant B is broken.
      • Sandbox Solution: We use labels in Prometheus (tenant_id, model). This allows creating dedicated Grafana dashboards per tenant. The Histogram context manager (with INFERENCE_LATENCY...) automatically captures the duration of the actual inference, excluding authentication/network overhead, giving precise performance data.
    3. Throttling (The "Leaky Bucket")

      • Problem: A "Noisy Neighbor" (Tenant B sending infinite requests) can starve resources (GPU memory/CPU) from other tenants.
      • Sandbox Solution: The RateLimiter checks a counter in Redis (fast shared state) before allowing the request to hit the model. This is a Distributed Throttle, ensuring that even if you scale the API to 100 server instances, the limit is enforced globally across the cluster.

    Scaling to Physical Isolation

    If the isolation requirements are strict (e.g., HIPAA compliance where code execution must not share memory), you would move the InferenceSandbox logic into a Sidecar or Sandbox Service:

    1. Main API receives the request.
    2. It serializes the request and tenant credentials.
    3. It spins up a short-lived container (e.g., Firecracker microVM) or sends a gRPC request to a dedicated tenant-specific pod.
    4. That pod has the specific secrets mounted volume.
    5. The pod returns the result and is destroyed/wiped.

    The logical sandbox provided above is the standard architectural pattern for SaaS products balancing security, cost, and latency.