can you give me a step by step on how to setup a local ai assistant that mainly uses…
- User
can you give me a step by step on how to setup a local ai assistant that mainly uses voice to communicate. I have an rtx 3080 gpu and a 12600k intel cpu. I would prefer open source software/applications that run 100% locally and are private (no data sent to servers). The main use of this local ai model is to converse in french and other languages (english, etc.)
Assistant
Here's a step-by-step guide to set up a local multilingual voice-enabled AI assistant using open-source tools, optimized for your hardware and privacy requirements:
Step 1: System Requirements & Setup
-
OS: Use Ubuntu 22.04 LTS or Windows 11 (WSL2 recommended for Linux compatibility).
-
Drivers:
- Install NVIDIA drivers (version ≥ 535) and CUDA 12.1 toolkit.
- Verify with
nvidia-smiandnvcc --version.
-
Python Environment:
sudo apt update && sudo apt install python3-venv python3-pip python3 -m venv ~/ai-assistant source ~/ai-assistant/bin/activate
Step 2: Speech-to-Text (STT) - Whisper or Vosk
Option A: Whisper (Multilingual)
pip install git+https://github.com/openai/whisper.git pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118- Download a multilingual model:
mkdir -p ~/models/whisper && cd ~/models/whisper wget https://openaipublic.azureedge.net/main/whisper/models/3f86df7f491bbdbba9c4f5bb1d1a781f/model.pt - Test with French audio:
import whisper model = whisper.load_model("medium", device="cuda") result = model.transcribe("french_audio.wav", language="French") print(result["text"])
Option B: Vosk (Lighter, Multilingual)
pip install vosk wget https://alphacephei.com/vosk/models/model-fr-0.22.zip # French unzip model-fr-0.22.zip -d ~/models/vosk-fr
Step 3: Language Model (LLM) - Phi-3 or Llama 3
Option A: Phi-3 (Small, Efficient)
- Download from Hugging Face:
pip install transformers accelerate bitsandbytes - Run with GPU acceleration:
from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini", device_map="auto") tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini")
Option B: Llama 3 (8B, Quantized)
- Use llama.cpp for GPU inference:
git clone https://github.com/ggerganov/llama.cpp cd llama.cpp && make -j && pip install -e . - Convert Llama 3 to GGUF:
python convert_hf_to_gguf.py /path/to/llama3 --outfile ~/models/llama3.gguf - Run quantized model:
./main -m ~/models/llama3.gguf -ngl 100 # Use 100 layers on GPU
Step 4: Text-to-Speech (TTS) - Coqui TTS or VITS
pip install TTS # Download multilingual models: mkdir -p ~/models/tts cd ~/models/tts # French model: wget https://github.com/coqui-ai/TTS/releases/download/v0.13.0/model_file.ckpt- Example script:
from TTS.api import TTS tts = TTS(model_path="~/models/tts/model_file.ckpt", progress_bar=False) tts.tts_to_file(text="Bonjour!", file_path="output.wav")
Step 5: Integration Script
Create a Python script (
assistant.py) to link components:import whisper # Or vosk from transformers import pipeline # Or llama.cpp API from TTS.api import TTS import sounddevice as sd import numpy as np # STT: Replace with Vosk/Whisper def listen(): print("Listening...") # Capture audio and return transcribed text return "Bonjour" # LLM: Replace with Phi-3 or Llama def respond(text): print(f"User: {text}") return "Comment puis-je vous aider?" # TTS: Replace with Coqui/VITS def speak(text): print(f"Assistant: {text}") tts.tts_to_file(text=text, file_path="response.wav") # Play audio with `playsound` or `pydub` if __name__ == "__main__": while True: user_input = listen() response = respond(user_input) speak(response)
Step 6: Optional Enhancements
-
Wake Word Detection:
- Use Porcupine:
pip install pvporcupine
- Use Porcupine:
-
Translation Layer (if LLM lacks French fluency):
pip install transformers translator = pipeline("translation_fr_to_en", model="Helsinki-NLP/opus-mt-fr-en") -
Optimize Performance:
- Use
CUDA_LAUNCH_BLOCKING=1for deterministic execution. - Limit threads:
torch.set_num_threads(4).
- Use
Step 7: Privacy & Offline Mode
- Disable internet access for the assistant process:
sudo ufw deny out to any port 80,443 - Verify no external connections:
lsof -i -P | grep -i "listen"
Testing & Troubleshooting
- Test each component independently:
python3 -c "import torch; print(torch.cuda.is_available())" - Monitor GPU usage:
nvidia-smi --query-gpu=index,temperature.gpu,utilization.gpu --format=csv - Reduce model size if needed: Use 4-bit quantization for LLMs.
This setup ensures full privacy, multilingual support, and leverages your GPU for performance. Adjust model sizes based on VRAM constraints (e.g., use Phi-3 for lower VRAM).
-
- User
I would prefer an app that has a gui with these features integrated instead of separate downloads. Open source preferred
Assistant
Here's a GUI-based open-source solution that integrates all your requirements (voice interaction, multilingual support, local-only processing) in a single app. This setup prioritizes privacy and leverages your hardware:
Option 1: GPT4All (Desktop App with Voice Integration)
Features:
- Local LLMs (supports Phi-3, Llama 3, Mistral, etc.)
- Built-in Whisper for speech-to-text (STT)
- Text-to-speech (TTS) integration via Coqui or Piper
- Multilingual (French/English)
- Open-source (GitHub: gpt4all)
Steps to Set Up:
-
Download & Install:
- GPT4All Desktop App (Windows/macOS/Linux).
-
Install Models:
- Download a multilingual LLM:
- Phi-3 (small, fast):
phi-3-mini-128k-instruct-q4.gguf - Llama 3 (larger, multilingual):
Meta-Llama-3-8B-Instruct-Q4_K_M.gguf
- Phi-3 (small, fast):
- Place models in
C:\Users\<user>\AppData\Local\nomic.ai\GPT4All(Windows) or~/Library/Application Support/nomic.ai/GPT4All(macOS).
- Download a multilingual LLM:
-
Enable Speech-to-Text (Whisper):
- Install Whisper:
pip install git+https://github.com/openai/whisper.git - In GPT4All:
- Go to Settings > Speech Recognition.
- Select
Whisper (Local)and choose the multilingual model (tiny,base, orsmall).
- Install Whisper:
-
Enable Text-to-Speech (Coqui or Piper):
- Install Coqui TTS:
pip install TTS - Download a French TTS model:
wget https://github.com/coqui-ai/TTS/releases/download/v0.13.0/fr_FR-vits-maestro_16000.pth - In GPT4All:
- Go to Settings > Text-to-Speech.
- Select
Coqui TTSand load the French model.
- Install Coqui TTS:
-
Use the App:
- Click the microphone icon to speak in French/English.
- GPT4All will transcribe, generate responses, and read them aloud.
Option 2: Jan (Private AI Assistant with GUI)
Features:
- Local LLMs (supports GGUF/Llama models)
- Plugin-based architecture for STT/TTS
- Multilingual (requires model selection)
- Open-source (GitHub: jan-ai)
Steps to Set Up:
-
Download Jan:
- jan.ai (Windows/macOS/Linux).
-
Install Models:
- Download a multilingual GGUF model (e.g.,
llama3-8b-italian-french-english-q4.gguf). - Place it in Jan's
models/folder.
- Download a multilingual GGUF model (e.g.,
-
Add STT/TTS Plugins:
- Install Whisper and Coqui as above.
- Enable plugins in Jan's settings:
- STT: Use
Whisperplugin (select multilingual model). - TTS: Use
Coquiplugin (load French model).
- STT: Use
-
Chat with Voice:
- Use the microphone button to speak.
- Jan will respond in your selected language.
Option 3: Oobabooga’s Text Generation Web UI (With Extensions)
Features:
- Local LLMs (supports Llama 3, Phi-3, etc.)
- Web-based GUI (runs on localhost)
- Extensions for STT/TTS
- Open-source (GitHub: oobabooga/text-generation-webui)
Steps to Set Up:
-
Clone the Repo:
git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui -
Install Dependencies:
pip install -r requirements.txt -
Download a Multilingual Model:
- Example:
Llama-3-8B-French-English-Italian-GGUF.
- Example:
-
Run the UI:
python server.py --model <model-name> --listen- Access the UI at
http://localhost:7860.
- Access the UI at
-
Add Voice Support:
- Install Whisper extension:
pip install git+https://github.com/openai/whisper.git - Install Coqui TTS extension:
pip install TTS - Enable extensions in the UI under Extensions.
- Install Whisper extension:
-
Use Voice Chat:
- Click the microphone icon in the chat interface.
Key Considerations
Feature GPT4All Jan Oobabooga GUI ✅ ✅ ✅ (Web) Multilingual LLM ✅ ✅ ✅ Whisper Integration ✅ ✅ ✅ TTS (French) ✅ ✅ ✅ CUDA Support ✅ ✅ ✅
Troubleshooting Tips
-
Language Support:
- Ensure your LLM is trained on French (e.g.,
Llama-3-8B-French-English). - Use translation layers if needed (e.g.,
Helsinki-NLP/opus-mt-fr-en).
- Ensure your LLM is trained on French (e.g.,
-
Performance:
- For your RTX 3080 (10GB VRAM), use 4-bit quantized models (e.g.,
Q4_K_M). - Enable CUDA in settings (GPT4All uses
cuBLASfor GGUF).
- For your RTX 3080 (10GB VRAM), use 4-bit quantized models (e.g.,
-
Privacy:
- Disable internet access for the app process:
sudo ufw deny out to any app <app-name>
- Disable internet access for the app process:
Let me know if you need help choosing a specific model or configuring one of these apps! 🚀
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.