Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-4-20250514· AnthropicHugging Face LMArena Human Preference 140K

What are the meaning of --batch-size and --ubatch-size in llama.cpp? Please give a…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Amazon Novaamazon.nova-pro-v1:0’s answer.
  1. User

    What are the meaning of --batch-size and --ubatch-size in llama.cpp? Please give a relatively short description. I'll probably need some more background info later but then I will ask for it.

  2. Claude Assistant

    In llama.cpp:

    --batch-size (or -b): The number of tokens the model processes simultaneously during prompt evaluation. Larger values use more memory but can speed up processing of long prompts. Default is typically 512.

    --ubatch-size (or -ub): The physical batch size used for processing. This is the actual number of tokens processed together at the hardware level. It's often smaller than batch-size to manage memory usage. When ubatch-size < batch-size, llama.cpp automatically splits the work into smaller chunks.

    Think of batch-size as how many tokens you want to handle at once logically, while ubatch-size is how many you actually process at once physically due to memory constraints.

  3. User

    I have a relatively small GPU, a Radeon 760M iGPU with 8 CU. I do have plenty of fast ddr5 RAM. Could you give a suggestion for setting --batch-size and --ubatch-size ?

  4. Claude Assistant

    For a Radeon 760M iGPU with 8 CUs, I'd suggest starting with:

    --batch-size 256 --ubatch-size 64

    The 760M shares system memory, so while you have plenty of DDR5 RAM, the GPU compute units are the bottleneck. The small ubatch-size (64) prevents overwhelming the limited CUs, while the moderate batch-size (256) allows decent prompt processing efficiency.

    You can experiment from there:

    • If it runs smoothly, try increasing ubatch-size to 128
    • If you get memory errors or slowdowns, reduce ubatch-size to 32
    • For very long prompts, you might benefit from batch-size 512, but keep ubatch-size small

    Since you're using an iGPU, also consider setting --gpu-layers (or -ngl) to a lower value if you're loading a large model - sometimes splitting between CPU and iGPU gives better performance than forcing everything onto the weak GPU.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.