Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-flash-preview-04-17· GoogleHugging Face LMArena Human Preference 140K

I have an Ubuntu machine with NVIDIA GPU and CUDA / drivers installed (for example,…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with MiniMaxminimax-m1’s answer.
  1. User

    I have an Ubuntu machine with NVIDIA GPU and CUDA / drivers installed (for example, Ollama runs on the GPU as expected). I want to run llama.cpp, so I've downloaded the appropriate llama.cpp binaries. llama-cli runs and invokes the specified GGUF model, but it runs entirely on the CPU, even if I specify --n-gpu-layers 100. How can I troubleshoot and fix this?

  2. Gemini Assistant

    Okay, this is a very common issue when using pre-compiled llama.cpp binaries or sometimes even when building yourself if the GPU support isn't explicitly enabled.

    The most likely reason --n-gpu-layers is ignored and it runs entirely on the CPU is that the llama.cpp binary you are using was not built with GPU acceleration enabled (specifically, CUDA/cuBLAS support for NVIDIA).

    Here's a step-by-step guide to troubleshoot and fix this:

    1. Verify Your System Setup (Quick Checks)

    You've mentioned Ollama works, which is a good sign, but let's double-check:

    • Check nvidia-smi: Run nvidia-smi in your terminal. Does it show your GPU, driver version, and potentially active processes (like Ollama if it's running)? This confirms the driver is loaded and the GPU is visible.
    • Check CUDA Version: Run nvcc --version. Does it show a CUDA toolkit version? (Note: llama.cpp often only needs the CUDA driver libraries at runtime, but having the toolkit installed is good for building).

    2. The Most Likely Culprit: llama.cpp Binary Not Built with GPU Support

    Downloaded binaries are often built with minimal dependencies (CPU only) or for specific configurations that might not match yours. The make or cmake command used to build llama.cpp must include flags to enable GPU acceleration.

    • Symptom Confirmation: If you run llama-cli (or the main example) with --n-gpu-layers <positive_number> and you do not see output lines like:
      • llm_load_tensors: offloading X layers to GPU
      • ggml_cuda_init: ...
      • Any other lines explicitly mentioning cuda or gpu during model loading... ...then the binary almost certainly lacks GPU support.

    3. Solution: Build llama.cpp from Source with CUDA Enabled

    This is the most reliable way to ensure you have a llama.cpp binary tailored to your system's CUDA installation.

    • Prerequisites:

      • Git (sudo apt update && sudo apt install git)
      • Build essentials (GCC, G++ etc. - sudo apt install build-essential cmake)
      • NVIDIA drivers and CUDA Toolkit (which you already have).
    • Steps:

      1. Clone the repository:

        git clone https://github.com/ggerganov/llama.cpp
        cd llama.cpp
        
      2. Build with CUDA (cuBLAS) support: llama.cpp uses make and environment variables or cmake flags to control the build process. The simplest way with make is typically:

        # Using make with environment variable (common and often sufficient)
        make clean # Start fresh
        make LLAMA_CUBLAS=1 -j$(nproc) # -j$(nproc) speeds up the build
        
        • LLAMA_CUBLAS=1 tells the build process to enable cuBLAS (CUDA Basic Linear Algebra Subprograms), which is what llama.cpp uses for GPU acceleration on NVIDIA cards.
        • -j$(nproc) tells make to use all available CPU cores for faster compilation.

        Alternatively, using CMake (more modern):

        # Using cmake (more standard, but slightly more involved)
        mkdir build
        cd build
        cmake .. -DLLAMA_CUBLAS=ON
        make -j$(nproc)
        # Binaries will be in ./build/bin/
        
        • -DLLAMA_CUBLAS=ON enables the cuBLAS backend.
      3. Verify the build: The build output should ideally mention detecting CUDA or cuBLAS.

      4. Find the new binaries:

        • If using the make LLAMA_CUBLAS=1 command in the main directory, the executables (main, llama-cli, server, etc.) will be created directly in the llama.cpp directory (e.g., ./main, ./llama-cli).
        • If using the cmake method, they will be in llama.cpp/build/bin/ (e.g., ./build/bin/main).

    4. Test the Newly Built Binary

    Now, use the binary you just built from source:

    # Example using the make-built binary (adjust path if using cmake build)
    ./llama-cli -m /path/to/your/model.gguf --n-gpu-layers 100 --prompt "Hello, world!" --n-predict 128 --verbose
    
    • Look for GPU initialization messages: When the model starts loading, carefully watch the output. You should now see lines indicating GPU initialization and layers being offloaded, like:

      ggml_init_cublas: found 1 CUDA devices:
        Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: 10000 MiB
      ...
      llm_load_tensors: offloading 32 layers to GPU
      

      (The exact output may vary slightly depending on your GPU and llama.cpp version).

    • Check nvidia-smi again: While the model is loading and running, run nvidia-smi in another terminal. You should see the memory usage of your GPU increase significantly as the model layers are loaded onto it, and you might see a process related to the llama-cli or main executable.

    5. Troubleshooting if Building Doesn't Work

    If building with LLAMA_CUBLAS=1 still doesn't result in GPU support (no CUDA messages during build or runtime):

    • Check Build Output: Look for errors or warnings during the make or cmake step related to CUDA, cuBLAS, or missing libraries.
    • CUDA Library Paths: Ensure your system can find the CUDA libraries. Sometimes you might need to set LD_LIBRARY_PATH. The build process usually handles this if CUDA is installed in standard locations (/usr/local/cuda).
    • Driver/Toolkit Mismatch: While less likely if Ollama works, ensure your NVIDIA driver version is compatible with the CUDA toolkit version llama.cpp is trying to use (usually the one nvcc --version points to, or system-wide libraries).
    • Permissions: Ensure your user has permission to access the GPU devices (/dev/nvidia*). This is usually set up correctly by the driver installer.
    • Dependencies: Check if any other required libraries for the CUDA backend are missing.

    Summary of Most Likely Cause:

    The downloaded binary didn't have CUDA support compiled in. Building llama.cpp from source with make LLAMA_CUBLAS=1 (or the equivalent CMake command) is the standard solution to enable GPU acceleration on NVIDIA cards.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.