how to configure Ollama Docker to run fast on CPU
- User
how to configure Ollama Docker to run fast on CPU
Assistant
To optimize Ollama Docker for CPU performance, follow these steps:
1. Basic Docker Setup
docker run -d -v ollama_data:/root/.ollama -p 11434:11434 --name ollama ollama/ollama2. CPU-Specific Optimizations
Add these environment variables to your Docker command:
docker run -d \ -v ollama_data:/root/.ollama \ -p 11434:11434 \ --name ollama \ -e OLLAMA_NUM_THREADS=8 \ # Set to physical cores (not hyperthreads) -e OLLAMA_BLAS=1 \ # Enable CPU acceleration -e OLLAMA_KEEP_ALIVE=5m \ # Reduce idle model unloading ollama/ollama3. Model Selection & Quantization
Pull a CPU-optimized model:
docker exec -it ollama ollama pull llama3.1:8b-gguf-q4_K_MUse smaller quantized models (4-bit/5-bit) like:
llama3.1:8b-gguf-q4_K_Mmistral:7b-gguf-q4_K_M
4. Memory Configuration
Add memory limits if needed:
docker run -d \ ... \ --memory=8g \ # Allocate RAM (adjust as needed) --memory-swap=12g # Allow swap if RAM < model size ollama/ollama5. Advanced CPU Tuning
For Intel CPUs with MKL:
-e OLLAMA_MKL_NUM_THREADS=8 \ -e OLLAMA_OPENBLAS_NUM_THREADS=8 \6. Verify Performance
Check CPU utilization:
docker statsKey Considerations:
- Thread Count: Match
OLLAMA_NUM_THREADSto physical cores - Quantization: Lower bit models (4-bit) run faster but less accurate
- Model Size: Start with 7B/8B parameter models
- Swap Space: Ensure sufficient swap if RAM < model size
- CPU Features: Enable AVX2/AVX-512 in Docker if available
Example Full Command:
docker run -d \ -v ollama_data:/root/.ollama \ -p 11434:11434 \ --name ollama \ --cpuset-cpus="0-7" \ # Pin to 8 physical cores -e OLLAMA_NUM_THREADS=8 \ -e OLLAMA_BLAS=1 \ -e OLLAMA_KEEP_ALIVE=5m \ --memory=8g \ --memory-swap=12g \ ollama/ollamaPost-Setup:
# Warm up the model docker exec -it ollama ollama run llama3.1:8b-gguf-q4_K_M "Hello world" # Check performance metrics docker logs ollamaNote: CPU inference will always be slower than GPU. For production use, consider cloud instances with higher CPU core counts (16+ cores recommended).
- User
what is the best way to create a list of values that constraint a variable in python
Assistant
To constrain a variable to a specific set of values in Python, here are the best approaches ranked by use case:
1. Enums (Best for Named Constants)
Use the
enummodule for type-safe, self-documenting allowed values:from enum import Enum class Color(Enum): RED = "red" GREEN = "green" BLUE = "blue" # Usage color: Color = Color.RED # Valid # color = "yellow" # Type checker/static analysis would flag thisBest for: Fixed set of named options (e.g., statuses, categories).
2. Literal Types (Python 3.8+ with Type Hints)
Use
Literalfor type hints and static validation:from typing import Literal AllowedValues = Literal["apple", "banana", "cherry"] def process(value: AllowedValues): print(value) process("apple") # Valid # process("orange") # Type checker (e.g., mypy) would flag thisBest for: Type-checked constraints with IDE support.
3. Set/List with Validation
Simple runtime checking for dynamic validation:
ALLOWED_VALUES = {"a", "b", "c"} # Use a set for O(1) lookups def validate(value): if value not in ALLOWED_VALUES: raise ValueError(f"{value} not allowed") return value validate("a") # Passes validate("d") # Raises ValueErrorBest for: Runtime validation of dynamic inputs (e.g., user input).
4. Pydantic Models (For Data Validation)
Use a validation library like Pydantic for complex constraints:
from pydantic import BaseModel, conlist class Item(BaseModel): tags: conlist(str, min_items=1, max_items=3) # List with length constraints # Usage Item(tags=["news", "sports"]) # Valid # Item(tags=[]) # Validation errorBest for: Data models with multiple constraints (e.g., APIs, configs).
5. Custom Class with Constants
Group allowed values in a class for readability:
class Status: ACTIVE = "active" PENDING = "pending" INACTIVE = "inactive" # Usage current_status = Status.ACTIVEBest for: Grouping related constants without enum overhead.
When to Use Which?
Method Use Case Pros Enum Named options, type safety Explicit, self-documenting Literal Static type checking IDE-friendly, lightweight Set/List + Check Runtime validation of dynamic data Simple, flexible Pydantic Complex data models with multiple rules Rich validation, JSON Schema Class Constants Grouping related values No dependencies
Example: Combining Approaches
from enum import Enum from typing import Literal # Enum for type safety class Mode(Enum): READ = "r" WRITE = "w" # Literal for function signatures def open_file(mode: Literal["r", "w"]): ... # Runtime check for external input ALLOWED_MODES = {"r", "w"} user_mode = input("Enter mode: ") if user_mode not in ALLOWED_MODES: raise ValueError("Invalid mode")Choose based on your need for explicitness, type safety, and validation complexity!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.