What are the meaning of --batch-size and --ubatch-size in llama.cpp? Please give a…
- User
What are the meaning of
--batch-sizeand--ubatch-sizein llama.cpp? Please give a relatively short description. I'll probably need some more background info later but then I will ask for it. Assistant
In llama.cpp:
--batch-size(or-b): The number of tokens the model processes simultaneously during prompt evaluation. Larger values use more memory but can speed up processing of long prompts. Default is typically 512.--ubatch-size(or-ub): The physical batch size used for processing. This is the actual number of tokens processed together at the hardware level. It's often smaller than batch-size to manage memory usage. When ubatch-size < batch-size, llama.cpp automatically splits the work into smaller chunks.Think of batch-size as how many tokens you want to handle at once logically, while ubatch-size is how many you actually process at once physically due to memory constraints.
- User
I have a relatively small GPU, a Radeon 760M iGPU with 8 CU. I do have plenty of fast ddr5 RAM. Could you give a suggestion for setting
--batch-sizeand--ubatch-size? Assistant
For a Radeon 760M iGPU with 8 CUs, I'd suggest starting with:
--batch-size 256 --ubatch-size 64The 760M shares system memory, so while you have plenty of DDR5 RAM, the GPU compute units are the bottleneck. The small ubatch-size (64) prevents overwhelming the limited CUs, while the moderate batch-size (256) allows decent prompt processing efficiency.
You can experiment from there:
- If it runs smoothly, try increasing ubatch-size to 128
- If you get memory errors or slowdowns, reduce ubatch-size to 32
- For very long prompts, you might benefit from batch-size 512, but keep ubatch-size small
Since you're using an iGPU, also consider setting
--gpu-layers(or-ngl) to a lower value if you're loading a large model - sometimes splitting between CPU and iGPU gives better performance than forcing everything onto the weak GPU.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.