Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning-pro-high· PerplexityHugging Face LMArena Search Arena 24K

Does koboldcpp support tensor parallelism?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning-pro-high’s answer.
  1. User

    Does koboldcpp support tensor parallelism?

  2. Perplexity Assistant

    Based on the search results, KoboldCpp primarily supports pipeline parallelism rather than tensor parallelism.

    KoboldCpp has demonstrated capabilities for multi-GPU utilization through pipeline parallelism, as evidenced by a GitHub issue discussion where users note that "when using several GPUs, the pipeline parallelism parameter always runs in two copies."[1] The developers have worked on optimizing this aspect, with changes to set the default number of copies to 4 when using parallel mode[1].

    KoboldCpp offers GPU acceleration through several features:

    • GPU processing for prompts using commands like --usecublas, --useclblast, or --usevulkan[3]
    • GPU layer offloading via --gpulayers which can significantly speed up inference by moving individual layers to run on the GPU[3]
    • Options to specify exactly how many layers to offload or let KoboldCpp guess with --gpulayers -1, though the automatic option "doesn't work accurately for multi-gpu setups"[3]

    In contrast, other tools in the ecosystem like TabbyAPI are mentioned as supporting tensor parallelism. One user specifically noted switching to "SillyTavern with TabbyAPI (after starting TabbyAPI using ./start.sh --tensor-parallel True command)" because of its tensor parallelism support.[2]

    While there is a partial mention of tensor parallelism in relation to KoboldCpp in one search result ("To enable tensor parallelism, append --tensor...")[10], the information is incomplete and doesn't provide definitive evidence of tensor parallelism support in KoboldCpp.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.