Skip to content
TheChatArchiveThe Chat Archive
MiniMaxminimax-m1· MiniMaxHugging Face LMArena Human Preference 140K

how to configure Ollama Docker to run fast on CPU

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o3-mini’s answer.
  1. User

    how to configure Ollama Docker to run fast on CPU

  2. MiniMax Assistant

    To optimize Ollama Docker for CPU performance, follow these steps:

    1. Basic Docker Setup

    docker run -d -v ollama_data:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
    

    2. CPU-Specific Optimizations

    Add these environment variables to your Docker command:

    docker run -d \
      -v ollama_data:/root/.ollama \
      -p 11434:11434 \
      --name ollama \
      -e OLLAMA_NUM_THREADS=8 \  # Set to physical cores (not hyperthreads)
      -e OLLAMA_BLAS=1 \        # Enable CPU acceleration
      -e OLLAMA_KEEP_ALIVE=5m \ # Reduce idle model unloading
      ollama/ollama
    

    3. Model Selection & Quantization

    Pull a CPU-optimized model:

    docker exec -it ollama ollama pull llama3.1:8b-gguf-q4_K_M
    

    Use smaller quantized models (4-bit/5-bit) like:

    • llama3.1:8b-gguf-q4_K_M
    • mistral:7b-gguf-q4_K_M

    4. Memory Configuration

    Add memory limits if needed:

    docker run -d \
      ... \
      --memory=8g \        # Allocate RAM (adjust as needed)
      --memory-swap=12g   # Allow swap if RAM < model size
      ollama/ollama
    

    5. Advanced CPU Tuning

    For Intel CPUs with MKL:

    -e OLLAMA_MKL_NUM_THREADS=8 \
    -e OLLAMA_OPENBLAS_NUM_THREADS=8 \
    

    6. Verify Performance

    Check CPU utilization:

    docker stats
    

    Key Considerations:

    1. Thread Count: Match OLLAMA_NUM_THREADS to physical cores
    2. Quantization: Lower bit models (4-bit) run faster but less accurate
    3. Model Size: Start with 7B/8B parameter models
    4. Swap Space: Ensure sufficient swap if RAM < model size
    5. CPU Features: Enable AVX2/AVX-512 in Docker if available

    Example Full Command:

    docker run -d \
      -v ollama_data:/root/.ollama \
      -p 11434:11434 \
      --name ollama \
      --cpuset-cpus="0-7" \     # Pin to 8 physical cores
      -e OLLAMA_NUM_THREADS=8 \
      -e OLLAMA_BLAS=1 \
      -e OLLAMA_KEEP_ALIVE=5m \
      --memory=8g \
      --memory-swap=12g \
      ollama/ollama
    

    Post-Setup:

    # Warm up the model
    docker exec -it ollama ollama run llama3.1:8b-gguf-q4_K_M "Hello world"
    
    # Check performance metrics
    docker logs ollama
    

    Note: CPU inference will always be slower than GPU. For production use, consider cloud instances with higher CPU core counts (16+ cores recommended).

  3. User

    what is the best way to create a list of values that constraint a variable in python

  4. MiniMax Assistant

    To constrain a variable to a specific set of values in Python, here are the best approaches ranked by use case:


    1. Enums (Best for Named Constants)

    Use the enum module for type-safe, self-documenting allowed values:

    from enum import Enum
    
    class Color(Enum):
        RED = "red"
        GREEN = "green"
        BLUE = "blue"
    
    # Usage
    color: Color = Color.RED  # Valid
    # color = "yellow"       # Type checker/static analysis would flag this
    

    Best for: Fixed set of named options (e.g., statuses, categories).


    2. Literal Types (Python 3.8+ with Type Hints)

    Use Literal for type hints and static validation:

    from typing import Literal
    
    AllowedValues = Literal["apple", "banana", "cherry"]
    
    def process(value: AllowedValues):
        print(value)
    
    process("apple")  # Valid
    # process("orange")  # Type checker (e.g., mypy) would flag this
    

    Best for: Type-checked constraints with IDE support.


    3. Set/List with Validation

    Simple runtime checking for dynamic validation:

    ALLOWED_VALUES = {"a", "b", "c"}  # Use a set for O(1) lookups
    
    def validate(value):
        if value not in ALLOWED_VALUES:
            raise ValueError(f"{value} not allowed")
        return value
    
    validate("a")  # Passes
    validate("d")  # Raises ValueError
    

    Best for: Runtime validation of dynamic inputs (e.g., user input).


    4. Pydantic Models (For Data Validation)

    Use a validation library like Pydantic for complex constraints:

    from pydantic import BaseModel, conlist
    
    class Item(BaseModel):
        tags: conlist(str, min_items=1, max_items=3)  # List with length constraints
    
    # Usage
    Item(tags=["news", "sports"])  # Valid
    # Item(tags=[])  # Validation error
    

    Best for: Data models with multiple constraints (e.g., APIs, configs).


    5. Custom Class with Constants

    Group allowed values in a class for readability:

    class Status:
        ACTIVE = "active"
        PENDING = "pending"
        INACTIVE = "inactive"
    
    # Usage
    current_status = Status.ACTIVE
    

    Best for: Grouping related constants without enum overhead.


    When to Use Which?

    MethodUse CasePros
    EnumNamed options, type safetyExplicit, self-documenting
    LiteralStatic type checkingIDE-friendly, lightweight
    Set/List + CheckRuntime validation of dynamic dataSimple, flexible
    PydanticComplex data models with multiple rulesRich validation, JSON Schema
    Class ConstantsGrouping related valuesNo dependencies

    Example: Combining Approaches

    from enum import Enum
    from typing import Literal
    
    # Enum for type safety
    class Mode(Enum):
        READ = "r"
        WRITE = "w"
    
    # Literal for function signatures
    def open_file(mode: Literal["r", "w"]):
        ...
    
    # Runtime check for external input
    ALLOWED_MODES = {"r", "w"}
    user_mode = input("Enter mode: ")
    if user_mode not in ALLOWED_MODES:
        raise ValueError("Invalid mode")
    

    Choose based on your need for explicitness, type safety, and validation complexity!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt