What do you say is the best way to get a LLM running locally - in my scenario? I do have…
- User
What do you say is the best way to get a LLM running locally - in my scenario? I do have an old single-core Thinkpad a Pi Zero 2 and a Pi 4B with 4GB. I don't want the LLM to then connect to any third-party server. I mainly wanna use it as "personal assistant"-type "chatting"
Assistant
Of course. This is a fantastic and increasingly common goal. Let's break down your scenario and find the best path forward.
First, a crucial bit of reality-setting to manage expectations: Running a modern Large Language Model (LLM) is computationally very demanding, primarily on RAM and secondarily on CPU/GPU processing power. The experience you're used to with services like ChatGPT runs on massive, expensive server farms.
Getting a "personal assistant-type" chat experience locally on your hardware is possible, but it will be slow and limited compared to commercial services. The key is to choose the right software and the right size of model.
Quick Assessment of Your Hardware
- Old Single-Core ThinkPad: This is unfortunately not a viable option. Modern LLMs require a significant amount of RAM (often more than 4GB just for the model) and multi-core processing to generate text at a usable speed. A single-core CPU would take many minutes to generate a single sentence.
- Raspberry Pi Zero 2: With only 512MB of RAM, this is also not a viable option. The operating system alone will use a large portion of that, leaving no room to load even the tiniest of language models.
- Raspberry Pi 4B (4GB): This is your winner. It's the only device in your list with enough RAM and a multi-core CPU to make this project feasible. All our focus will be on this device.
The Best Way: Your Raspberry Pi 4B
The strategy is to use highly efficient software designed specifically for running LLMs on CPUs, and to use a small, "quantized" model that can fit into your Pi's limited RAM.
Here is the best combination of tools for your scenario:
- The Engine:
llama.cppThis is a brilliant C/C++ project designed to run Llama-family models (and many others) with maximum efficiency on regular CPUs, including the ARM processor in your Pi. It's the gold standard for this kind of work. - The Model Format: GGUF (GPT-Generated Unified Format)
This is the model format that
llama.cppuses. It's a single-file format that contains everything needed to run the model. - The Magic Trick: Quantization This is the process of reducing the precision of the model's numbers (its "weights"), which drastically shrinks the file size and RAM usage. For your 4GB Pi, we'll need a heavily quantized model (e.g., 4-bit quantization). This slightly reduces the "intelligence" of the model, but it's the only way it will run.
- The Model Choice: A Small Model (1B to 3B parameters)
You cannot run a 7-billion or 13-billion parameter model. You need to look for smaller models.
- TinyLlama-1.1B: An excellent choice. It's specifically trained to be small but capable.
- Phi-2 (2.7B): A fantastic small model from Microsoft that punches well above its weight. It's very coherent for its size.
Step-by-Step Guide to Get It Running
This plan will give you a simple web interface on your Pi that you can access from any other computer on your network to chat with your local LLM.
Prerequisites on your Pi 4:
- A 64-bit version of Raspberry Pi OS (for better performance).
- A good power supply (to prevent under-voltage issues under load).
- A case with a fan or a heatsink. The Pi will be working very hard.
Step 1: Install
llama.cppOpen a terminal on your Raspberry Pi and run these commands:
# Update your system first sudo apt update && sudo apt upgrade -y # Install required tools sudo apt install -y git build-essential # Clone the llama.cpp repository git clone https://github.com/ggerganov/llama.cpp.git # Move into the directory and compile it cd llama.cpp makeThis will compile the software. It might take a few minutes.
Step 2: Download a Suitable Model
You can find thousands of GGUF models on Hugging Face. We'll download a 4-bit quantized version of Microsoft's Phi-2 model. It's about 1.6 GB, so it will fit in your RAM.
In your terminal (still in the
llama.cppdirectory is fine), run:# This downloads a small but very capable model wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q4_K_M.ggufStep 3: Run the LLM in Server Mode
The easiest way to get a "chat" interface is to use the simple web server built into
llama.cpp.Run this command from the
llama.cppdirectory:./server -m phi-2.Q4_K_M.gguf -c 2048 --host 0.0.0.0./server: Runs the server program you just compiled.-m phi-2.Q4_K_M.gguf: Specifies the model file to use.-c 2048: Sets the "context size" (like the model's short-term memory). 2048 is a good starting point.--host 0.0.0.0: Allows you to access the server from other devices on your local network.
Step 4: Chat with Your LLM!
- Find your Raspberry Pi's IP address. You can do this by opening a new terminal window on the Pi and typing
hostname -I. It will look something like192.168.1.55. - On your main computer (or a phone), open a web browser.
- Go to the address:
http://<YOUR_PI_IP_ADDRESS>:8080(e.g.,http://192.168.1.55:8080).
You will see a simple, clean webpage that provides a chat interface directly to your locally-running LLM.
What to Expect (Performance)
- Speed: It will not be instant. You will see the words appear one by one. Expect a speed of around 1-4 tokens (words/parts of words) per second. It will feel like chatting with a very slow, methodical typist.
- Quality: The Phi-2 model is surprisingly good for its size. It can follow instructions, answer questions, summarize text, and carry on a decent conversation. It will not be as creative, nuanced, or factually accurate as GPT-4, and it may sometimes get confused or repeat itself.
This is the absolute best way to achieve your goal with the hardware you have. It's a fantastic project and a great way to learn about how these models work under the hood.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.