I have an Ubuntu machine with NVIDIA GPU and CUDA / drivers installed (for example,…
- User
I have an Ubuntu machine with NVIDIA GPU and CUDA / drivers installed (for example, Ollama runs on the GPU as expected). I want to run llama.cpp, so I've downloaded the appropriate llama.cpp binaries.
llama-cliruns and invokes the specified GGUF model, but it runs entirely on the CPU, even if I specify--n-gpu-layers 100. How can I troubleshoot and fix this? Assistant
Okay, this is a very common issue when using pre-compiled
llama.cppbinaries or sometimes even when building yourself if the GPU support isn't explicitly enabled.The most likely reason
--n-gpu-layersis ignored and it runs entirely on the CPU is that thellama.cppbinary you are using was not built with GPU acceleration enabled (specifically, CUDA/cuBLAS support for NVIDIA).Here's a step-by-step guide to troubleshoot and fix this:
1. Verify Your System Setup (Quick Checks)
You've mentioned Ollama works, which is a good sign, but let's double-check:
- Check
nvidia-smi: Runnvidia-smiin your terminal. Does it show your GPU, driver version, and potentially active processes (like Ollama if it's running)? This confirms the driver is loaded and the GPU is visible. - Check CUDA Version: Run
nvcc --version. Does it show a CUDA toolkit version? (Note:llama.cppoften only needs the CUDA driver libraries at runtime, but having the toolkit installed is good for building).
2. The Most Likely Culprit:
llama.cppBinary Not Built with GPU SupportDownloaded binaries are often built with minimal dependencies (CPU only) or for specific configurations that might not match yours. The
makeorcmakecommand used to buildllama.cppmust include flags to enable GPU acceleration.- Symptom Confirmation: If you run
llama-cli(or themainexample) with--n-gpu-layers <positive_number>and you do not see output lines like:llm_load_tensors: offloading X layers to GPUggml_cuda_init: ...- Any other lines explicitly mentioning
cudaorgpuduring model loading... ...then the binary almost certainly lacks GPU support.
3. Solution: Build
llama.cppfrom Source with CUDA EnabledThis is the most reliable way to ensure you have a
llama.cppbinary tailored to your system's CUDA installation.-
Prerequisites:
- Git (
sudo apt update && sudo apt install git) - Build essentials (GCC, G++ etc. -
sudo apt install build-essential cmake) - NVIDIA drivers and CUDA Toolkit (which you already have).
- Git (
-
Steps:
-
Clone the repository:
git clone https://github.com/ggerganov/llama.cpp cd llama.cpp -
Build with CUDA (cuBLAS) support:
llama.cppusesmakeand environment variables orcmakeflags to control the build process. The simplest way withmakeis typically:# Using make with environment variable (common and often sufficient) make clean # Start fresh make LLAMA_CUBLAS=1 -j$(nproc) # -j$(nproc) speeds up the buildLLAMA_CUBLAS=1tells the build process to enable cuBLAS (CUDA Basic Linear Algebra Subprograms), which is whatllama.cppuses for GPU acceleration on NVIDIA cards.-j$(nproc)tellsmaketo use all available CPU cores for faster compilation.
Alternatively, using CMake (more modern):
# Using cmake (more standard, but slightly more involved) mkdir build cd build cmake .. -DLLAMA_CUBLAS=ON make -j$(nproc) # Binaries will be in ./build/bin/-DLLAMA_CUBLAS=ONenables the cuBLAS backend.
-
Verify the build: The build output should ideally mention detecting CUDA or cuBLAS.
-
Find the new binaries:
- If using the
make LLAMA_CUBLAS=1command in the main directory, the executables (main,llama-cli,server, etc.) will be created directly in thellama.cppdirectory (e.g.,./main,./llama-cli). - If using the
cmakemethod, they will be inllama.cpp/build/bin/(e.g.,./build/bin/main).
- If using the
-
4. Test the Newly Built Binary
Now, use the binary you just built from source:
# Example using the make-built binary (adjust path if using cmake build) ./llama-cli -m /path/to/your/model.gguf --n-gpu-layers 100 --prompt "Hello, world!" --n-predict 128 --verbose-
Look for GPU initialization messages: When the model starts loading, carefully watch the output. You should now see lines indicating GPU initialization and layers being offloaded, like:
ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: 10000 MiB ... llm_load_tensors: offloading 32 layers to GPU(The exact output may vary slightly depending on your GPU and
llama.cppversion). -
Check
nvidia-smiagain: While the model is loading and running, runnvidia-smiin another terminal. You should see the memory usage of your GPU increase significantly as the model layers are loaded onto it, and you might see a process related to thellama-cliormainexecutable.
5. Troubleshooting if Building Doesn't Work
If building with
LLAMA_CUBLAS=1still doesn't result in GPU support (no CUDA messages during build or runtime):- Check Build Output: Look for errors or warnings during the
makeorcmakestep related to CUDA, cuBLAS, or missing libraries. - CUDA Library Paths: Ensure your system can find the CUDA libraries. Sometimes you might need to set
LD_LIBRARY_PATH. The build process usually handles this if CUDA is installed in standard locations (/usr/local/cuda). - Driver/Toolkit Mismatch: While less likely if Ollama works, ensure your NVIDIA driver version is compatible with the CUDA toolkit version
llama.cppis trying to use (usually the onenvcc --versionpoints to, or system-wide libraries). - Permissions: Ensure your user has permission to access the GPU devices (
/dev/nvidia*). This is usually set up correctly by the driver installer. - Dependencies: Check if any other required libraries for the CUDA backend are missing.
Summary of Most Likely Cause:
The downloaded binary didn't have CUDA support compiled in. Building
llama.cppfrom source withmake LLAMA_CUBLAS=1(or the equivalent CMake command) is the standard solution to enable GPU acceleration on NVIDIA cards.- Check
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.