Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

can you give me a step by step on how to setup a local ai assistant that mainly uses…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-sonnet-20241022’s answer.
  1. User

    can you give me a step by step on how to setup a local ai assistant that mainly uses voice to communicate. I have an rtx 3080 gpu and a 12600k intel cpu. I would prefer open source software/applications that run 100% locally and are private (no data sent to servers). The main use of this local ai model is to converse in french and other languages (english, etc.)

  2. Qwen Assistant

    Here's a step-by-step guide to set up a local multilingual voice-enabled AI assistant using open-source tools, optimized for your hardware and privacy requirements:


    Step 1: System Requirements & Setup

    1. OS: Use Ubuntu 22.04 LTS or Windows 11 (WSL2 recommended for Linux compatibility).

    2. Drivers:

      • Install NVIDIA drivers (version ≥ 535) and CUDA 12.1 toolkit.
      • Verify with nvidia-smi and nvcc --version.
    3. Python Environment:

      sudo apt update && sudo apt install python3-venv python3-pip
      python3 -m venv ~/ai-assistant
      source ~/ai-assistant/bin/activate
      

    Step 2: Speech-to-Text (STT) - Whisper or Vosk

    Option A: Whisper (Multilingual)

    pip install git+https://github.com/openai/whisper.git
    pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
    
    • Download a multilingual model:
      mkdir -p ~/models/whisper && cd ~/models/whisper
      wget https://openaipublic.azureedge.net/main/whisper/models/3f86df7f491bbdbba9c4f5bb1d1a781f/model.pt
      
    • Test with French audio:
      import whisper
      model = whisper.load_model("medium", device="cuda")
      result = model.transcribe("french_audio.wav", language="French")
      print(result["text"])
      

    Option B: Vosk (Lighter, Multilingual)

    pip install vosk
    wget https://alphacephei.com/vosk/models/model-fr-0.22.zip  # French
    unzip model-fr-0.22.zip -d ~/models/vosk-fr
    

    Step 3: Language Model (LLM) - Phi-3 or Llama 3

    Option A: Phi-3 (Small, Efficient)

    • Download from Hugging Face:
      pip install transformers accelerate bitsandbytes
      
    • Run with GPU acceleration:
      from transformers import AutoModelForCausalLM, AutoTokenizer
      model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini", device_map="auto")
      tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini")
      

    Option B: Llama 3 (8B, Quantized)

    1. Use llama.cpp for GPU inference:
      git clone https://github.com/ggerganov/llama.cpp
      cd llama.cpp && make -j && pip install -e .
      
    2. Convert Llama 3 to GGUF:
      python convert_hf_to_gguf.py /path/to/llama3 --outfile ~/models/llama3.gguf
      
    3. Run quantized model:
      ./main -m ~/models/llama3.gguf -ngl 100  # Use 100 layers on GPU
      

    Step 4: Text-to-Speech (TTS) - Coqui TTS or VITS

    pip install TTS
    # Download multilingual models:
    mkdir -p ~/models/tts
    cd ~/models/tts
    # French model:
    wget https://github.com/coqui-ai/TTS/releases/download/v0.13.0/model_file.ckpt
    
    • Example script:
      from TTS.api import TTS
      tts = TTS(model_path="~/models/tts/model_file.ckpt", progress_bar=False)
      tts.tts_to_file(text="Bonjour!", file_path="output.wav")
      

    Step 5: Integration Script

    Create a Python script (assistant.py) to link components:

    import whisper  # Or vosk
    from transformers import pipeline  # Or llama.cpp API
    from TTS.api import TTS
    import sounddevice as sd
    import numpy as np
    
    # STT: Replace with Vosk/Whisper
    def listen():
        print("Listening...")
        # Capture audio and return transcribed text
        return "Bonjour"
    
    # LLM: Replace with Phi-3 or Llama
    def respond(text):
        print(f"User: {text}")
        return "Comment puis-je vous aider?"
    
    # TTS: Replace with Coqui/VITS
    def speak(text):
        print(f"Assistant: {text}")
        tts.tts_to_file(text=text, file_path="response.wav")
        # Play audio with `playsound` or `pydub`
    
    if __name__ == "__main__":
        while True:
            user_input = listen()
            response = respond(user_input)
            speak(response)
    

    Step 6: Optional Enhancements

    1. Wake Word Detection:

    2. Translation Layer (if LLM lacks French fluency):

      pip install transformers
      translator = pipeline("translation_fr_to_en", model="Helsinki-NLP/opus-mt-fr-en")
      
    3. Optimize Performance:

      • Use CUDA_LAUNCH_BLOCKING=1 for deterministic execution.
      • Limit threads: torch.set_num_threads(4).

    Step 7: Privacy & Offline Mode

    • Disable internet access for the assistant process:
      sudo ufw deny out to any port 80,443
      
    • Verify no external connections:
      lsof -i -P | grep -i "listen"
      

    Testing & Troubleshooting

    1. Test each component independently:
      python3 -c "import torch; print(torch.cuda.is_available())"
      
    2. Monitor GPU usage:
      nvidia-smi --query-gpu=index,temperature.gpu,utilization.gpu --format=csv
      
    3. Reduce model size if needed: Use 4-bit quantization for LLMs.

    This setup ensures full privacy, multilingual support, and leverages your GPU for performance. Adjust model sizes based on VRAM constraints (e.g., use Phi-3 for lower VRAM).

  3. User

    I would prefer an app that has a gui with these features integrated instead of separate downloads. Open source preferred

  4. Qwen Assistant

    Here's a GUI-based open-source solution that integrates all your requirements (voice interaction, multilingual support, local-only processing) in a single app. This setup prioritizes privacy and leverages your hardware:


    Option 1: GPT4All (Desktop App with Voice Integration)

    Features:

    • Local LLMs (supports Phi-3, Llama 3, Mistral, etc.)
    • Built-in Whisper for speech-to-text (STT)
    • Text-to-speech (TTS) integration via Coqui or Piper
    • Multilingual (French/English)
    • Open-source (GitHub: gpt4all)

    Steps to Set Up:

    1. Download & Install:

    2. Install Models:

      • Download a multilingual LLM:
        • Phi-3 (small, fast): phi-3-mini-128k-instruct-q4.gguf
        • Llama 3 (larger, multilingual): Meta-Llama-3-8B-Instruct-Q4_K_M.gguf
      • Place models in C:\Users\<user>\AppData\Local\nomic.ai\GPT4All (Windows) or ~/Library/Application Support/nomic.ai/GPT4All (macOS).
    3. Enable Speech-to-Text (Whisper):

      • Install Whisper:
        pip install git+https://github.com/openai/whisper.git
        
      • In GPT4All:
        • Go to Settings > Speech Recognition.
        • Select Whisper (Local) and choose the multilingual model (tiny, base, or small).
    4. Enable Text-to-Speech (Coqui or Piper):

      • Install Coqui TTS:
        pip install TTS
        
      • Download a French TTS model:
        wget https://github.com/coqui-ai/TTS/releases/download/v0.13.0/fr_FR-vits-maestro_16000.pth
        
      • In GPT4All:
        • Go to Settings > Text-to-Speech.
        • Select Coqui TTS and load the French model.
    5. Use the App:

      • Click the microphone icon to speak in French/English.
      • GPT4All will transcribe, generate responses, and read them aloud.

    Option 2: Jan (Private AI Assistant with GUI)

    Features:

    • Local LLMs (supports GGUF/Llama models)
    • Plugin-based architecture for STT/TTS
    • Multilingual (requires model selection)
    • Open-source (GitHub: jan-ai)

    Steps to Set Up:

    1. Download Jan:

      • jan.ai (Windows/macOS/Linux).
    2. Install Models:

      • Download a multilingual GGUF model (e.g., llama3-8b-italian-french-english-q4.gguf).
      • Place it in Jan's models/ folder.
    3. Add STT/TTS Plugins:

      • Install Whisper and Coqui as above.
      • Enable plugins in Jan's settings:
        • STT: Use Whisper plugin (select multilingual model).
        • TTS: Use Coqui plugin (load French model).
    4. Chat with Voice:

      • Use the microphone button to speak.
      • Jan will respond in your selected language.

    Option 3: Oobabooga’s Text Generation Web UI (With Extensions)

    Features:

    • Local LLMs (supports Llama 3, Phi-3, etc.)
    • Web-based GUI (runs on localhost)
    • Extensions for STT/TTS
    • Open-source (GitHub: oobabooga/text-generation-webui)

    Steps to Set Up:

    1. Clone the Repo:

      git clone https://github.com/oobabooga/text-generation-webui
      cd text-generation-webui
      
    2. Install Dependencies:

      pip install -r requirements.txt
      
    3. Download a Multilingual Model:

      • Example: Llama-3-8B-French-English-Italian-GGUF.
    4. Run the UI:

      python server.py --model <model-name> --listen
      
      • Access the UI at http://localhost:7860.
    5. Add Voice Support:

      • Install Whisper extension:
        pip install git+https://github.com/openai/whisper.git
        
      • Install Coqui TTS extension:
        pip install TTS
        
      • Enable extensions in the UI under Extensions.
    6. Use Voice Chat:

      • Click the microphone icon in the chat interface.

    Key Considerations

    FeatureGPT4AllJanOobabooga
    GUI✅✅✅ (Web)
    Multilingual LLM✅✅✅
    Whisper Integration✅✅✅
    TTS (French)✅✅✅
    CUDA Support✅✅✅

    Troubleshooting Tips

    1. Language Support:

      • Ensure your LLM is trained on French (e.g., Llama-3-8B-French-English).
      • Use translation layers if needed (e.g., Helsinki-NLP/opus-mt-fr-en).
    2. Performance:

      • For your RTX 3080 (10GB VRAM), use 4-bit quantized models (e.g., Q4_K_M).
      • Enable CUDA in settings (GPT4All uses cuBLAS for GGUF).
    3. Privacy:

      • Disable internet access for the app process:
        sudo ufw deny out to any app <app-name>
        

    Let me know if you need help choosing a specific model or configuring one of these apps! 🚀

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.