Does koboldcpp support tensor parallelism?
- User
Does koboldcpp support tensor parallelism?
Assistant
Based on the search results, KoboldCpp primarily supports pipeline parallelism rather than tensor parallelism.
KoboldCpp has demonstrated capabilities for multi-GPU utilization through pipeline parallelism, as evidenced by a GitHub issue discussion where users note that "when using several GPUs, the pipeline parallelism parameter always runs in two copies."[1] The developers have worked on optimizing this aspect, with changes to set the default number of copies to 4 when using parallel mode[1].
KoboldCpp offers GPU acceleration through several features:
- GPU processing for prompts using commands like
--usecublas,--useclblast, or--usevulkan[3] - GPU layer offloading via
--gpulayerswhich can significantly speed up inference by moving individual layers to run on the GPU[3] - Options to specify exactly how many layers to offload or let KoboldCpp guess with
--gpulayers -1, though the automatic option "doesn't work accurately for multi-gpu setups"[3]
In contrast, other tools in the ecosystem like TabbyAPI are mentioned as supporting tensor parallelism. One user specifically noted switching to "SillyTavern with TabbyAPI (after starting TabbyAPI using ./start.sh --tensor-parallel True command)" because of its tensor parallelism support.[2]
While there is a partial mention of tensor parallelism in relation to KoboldCpp in one search result ("To enable tensor parallelism, append --tensor...")[10], the information is incomplete and doesn't provide definitive evidence of tensor parallelism support in KoboldCpp.
- GPU processing for prompts using commands like
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.