What are my alternatives when it comes to running LLMs locally on Linux
- User
What are my alternatives when it comes to running LLMs locally on Linux
Assistant
For running LLMs locally on Linux, you have several good options:
Ollama is one of the most popular choices - it's a command-line tool that makes it easy to download and run models like Llama, Mistral, and Gemma. It handles model management and provides a simple API.
LM Studio offers a user-friendly GUI for downloading and running various open models. While primarily known for its Windows/Mac versions, it does have Linux support.
llama.cpp is a lightweight C++ implementation that's highly optimized for CPU inference. It supports many model formats and has excellent performance even without a GPU. You can use it directly or through various frontends.
text-generation-webui (often called "oobabooga") is a Gradio-based interface that supports multiple model backends and formats. It's feature-rich with options for different sampling methods, character roleplay, and extensions.
vLLM is optimized for high-throughput serving and is particularly good if you need to handle multiple concurrent requests. It requires a GPU but offers excellent performance.
LocalAI provides an OpenAI-compatible API for local models, making it easy to use with existing tools that expect OpenAI's API format.
Hugging Face Transformers gives you the most flexibility if you're comfortable with Python - you can load and run models directly with fine-grained control over the inference process.
The best choice depends on your hardware (especially whether you have a capable GPU), technical comfort level, and specific use case. For beginners, Ollama or LM Studio are often the easiest starting points.
- User
I have a strong and cutting edge PC with an 5090 installed. I want to run two models in parallel (one small for embeddings one larger for inference) I will have interact with these models through an API
Assistant
With a 5090 (impressive hardware!), you have excellent options for running dual models with API access. Here are the best approaches for your use case:
vLLM would be my top recommendation for your setup. It's specifically designed for high-performance serving with API endpoints and can handle multiple models simultaneously. You can run two vLLM instances on different ports - one for your embedding model and one for your inference model. It supports tensor parallelism and has excellent memory management for the 5090.
LocalAI is another strong contender since it's built specifically for running multiple models behind an OpenAI-compatible API. You can configure it to load both models and route requests appropriately. This is particularly convenient if your applications already use OpenAI's API format.
Ray Serve with Hugging Face models gives you production-grade serving capabilities. It can efficiently manage multiple models on your GPU with automatic batching and resource allocation. Ray handles the complexity of parallel model serving very well.
TGI (Text Generation Inference) by Hugging Face is optimized for serving and supports running multiple model instances. It provides OpenAI-compatible endpoints and has features like continuous batching and tensor parallelism that would work well with your 5090.
Custom solution with FastAPI + Hugging Face Transformers or llama.cpp bindings gives you maximum control. You could load both models into GPU memory and create custom endpoints for each. This requires more setup but offers complete flexibility.
For embeddings specifically, you might also consider using Infinity, which is purpose-built for serving embedding models with high throughput.
Given your powerful hardware and dual-model requirement, I'd lean toward vLLM or TGI for the best performance, with LocalAI as a simpler alternative if you prefer easier setup over maximum optimization.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.