Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· openaiHugging Face Suzhen/CodeChat

How to use openai-whisper python package for mic recording recognition on the fly? My…

  1. User

    How to use openai-whisper python package for mic recording recognition on the fly? My goal is to automatically start script on boot (Ubuntu), check availability of input-output audio devices and start recording with stream through ASR (later i want to add commands recognition and "skills", but for now let's just write transcription with timestamps into text file). Basic example from GitHub: ```python import whisper

    model = whisper.load_model("base") result = model.transcribe("audio.mp3") print(result["text"])```

  2. Assistant

    Creating a script to recognize speech from the microphone in real-time using OpenAI's Whisper and then automatically running it on boot in an Ubuntu system involves several steps. Below, I'll guide you through setting up a basic example that captures audio from the microphone, transcribes it using Whisper, and writes the transcription with timestamps into a text file. Additionally, I'll show you how to ensure it runs at startup.

    Step 1: Install Necessary Packages

    First, ensure you have Python 3.6 or later installed. Then, install OpenAI's Whisper and other necessary packages. You might also need sounddevice to capture audio from the microphone in real-time and numpy for handling audio data.

    pip install whisper sounddevice numpy
    

    Step 2: Create the Python Script

    Create a Python script, whisper_mic.py, to capture audio from the microphone and transcribe it in real-time. This will be a simplified example; you might need to refine it based on your requirements.

    import numpy as np
    import sounddevice as sd
    import whisper
    import time
    import json
    
    # Load Whisper model
    model = whisper.load_model("base")
    
    # Define callback function to stream audio from mic
    def callback(indata, frames, time, status):
        # Check if there's an error capturing audio
        if status:
            print(status, file=sys.stderr)
        # Convert audio data to proper format
        audio_numpy = np.array(indata[:,0], dtype=np.float32)
        result = model.transcribe(audio_numpy, verbose=True)
        print(result["text"])
        # Optionally, save the transcription with timestamps into a text file
        with open("transcription.txt", "a") as file:
            timestamp = time.ctime()
            transcription = result["text"]
            file.write(f"{timestamp}: {transcription}\n")
    
    # Start streaming from microphone
    with sd.InputStream(callback=callback, dtype='float32', channels=1, samplerate=16000):
        print("Recording...")
        while True:
            # Keep the script alive
            time.sleep(0.1)
    

    Step 3: Test the Script

    Before setting it to start at boot, manually test the script to ensure it works as expected:

    python whisper_mic.py
    

    Step 4: Running the Script on Boot

    To run the script at startup, you can use systemd in Ubuntu.

    1. Create a systemd service file for your script:
    sudo nano /etc/systemd/system/whisper_mic.service
    
    1. Add the following content to the service file:
    [Unit]
    Description=Whisper Microphone Transcription
    After=multi-user.target
    
    [Service]
    Type=idle
    User=your_username
    ExecStart=/usr/bin/python3 /path/to/whisper_mic.py
    Restart=always
    
    [Install]
    WantedBy=multi-user.target
    

    Replace your_username with your actual username and /path/to/whisper_mic.py with the full path to your script.

    1. Enable and start the service:
    sudo systemctl daemon-reload
    sudo systemctl enable whisper_mic.service
    sudo systemctl start whisper_mic.service
    
    1. Check the status of your service:
    sudo systemctl status whisper_mic.service
    

    This is a basic framework. Depending on your exact requirements, such as handling more advanced audio processing, integrating command recognition, or developing specific "skills," you will need to build upon this foundation. Additionally, always monitor the performance and adjust the model or resources as necessary, especially when running heavy workloads like real-time audio processing and transcription on boot-up.

  3. User

    I might want to add noise reduction. Noise profile is more or less constant (mostly from working desktop itself - coolers can be heard). Not sure if we should discard it entirely (noise level can be used to estimate PC load), because anomaly detection is also considered.

  4. Assistant

    Adding noise reduction to your real-time transcription system can significantly improve speech recognition accuracy, especially in environments with constant background noise, such as cooler fans from a desktop. The Whisper model is quite robust in handling various noise conditions, but pre-processing the audio to reduce noise can still be beneficial.

    Given your interest in potentially leveraging noise levels for anomaly detection or estimating PC load, it might be useful to take a two-pronged approach: perform noise reduction for the purpose of improving speech recognition, while also analyzing the noise level to extract useful information.

    Step 1: Noise Reduction

    To perform noise reduction, you can use the noisereduce library, which is a Python package that can automatically reduce noise from audio.

    1. Install the library:
    pip install noisereduce
    
    1. Modify the callback function to incorporate noise reduction:

    First, record a short segment of your background noise without speech and save it as a .wav file. You'll use this as your noise profile.

    Modify your callback function in the whisper_mic.py script to use noisereduce:

    import noisereduce as nr
    # Load your noise profile
    noise_clip = np.loadtxt('noise_profile.npy')
    
    def callback(indata, frames, time, status):
        if status:
            print(status, file=sys.stderr)
        # Convert to numpy array and reduce noise
        audio_numpy = np.array(indata[:,0], dtype=np.float32)
        reduced_noise_audio = nr.reduce_noise(y=audio_numpy, sr=16000, y_noise=noise_clip, use_tensorflow=False)
        
        result = model.transcribe(reduced_noise_audio, verbose=True)
        print(result["text"])
        
        with open("transcription.txt", "a") as file:
            timestamp = time.ctime()
            transcription = result["text"]
            file.write(f"{timestamp}: {transcription}\n")
    

    Ensure you have created a noise_profile.npy by recording the background noise and saving the NumPy array using np.savetxt('noise_profile.npy', noise_data).

    Step 2: Noise Level Analysis

    If you're interested in using the noise level for estimating PC load or anomaly detection, you can calculate the sound level from the raw audio data and perform analysis or log these levels for review.

    1. Calculate the sound level:

    Add a function to calculate the sound level (dB) from the audio:

    def calculate_sound_level(audio):
        # Convert audio to dB
        sound_level = 20 * np.log10(np.std(audio))
        return sound_level
    
    1. Integrate the sound level calculation:

    Integrate this function into your workflow, possibly before noise reduction, to analyze or log the background noise level for further use in anomaly detection or PC load estimation.

        ...
        audio_numpy = np.array(indata[:,0], dtype=np.float32)
        sound_level = calculate_sound_level(audio_numpy)
        # You can log or analyze this sound_level as needed
        ...
    

    Considerations

    • Real-time Processing: All modifications should ensure that the real-time processing demands do not exceed your system's capabilities. Noise reduction and additional analysis can be computationally expensive.
    • Anomaly Detection: If you're interested in anomaly detection, consider establishing a baseline sound level during typical operating conditions. Significant deviations from this baseline could be used to trigger alerts or further investigation.
    • Adaptive Noise Cancellation: For more advanced scenarios, consider looking into adaptive noise cancellation techniques that continuously learn and adjust to the background noise in real-time.

    This approach allows you to not only improve the accuracy of your speech recognition by reducing noise but also gives you the flexibility to analyze the environment's acoustic characteristics for other applications.

  5. User

    After ASR I want to add LLM (e.g. LLaMA, from huggingface transformers with bitsandbytes 8bit) to process text and identify task. E.g. write notes, check news/weather, calendar integration, etc (focus on self development and reducing cognitive load).

  6. Assistant

    Integrating a Large Language Model (LLM) like LLaMA (from Hugging Face Transformers) into your speech recognition system to process text and identify tasks is an excellent way to build a sophisticated, personal assistant-like application. Utilizing bitsandbytes for 8-bit quantization can help make the model more memory efficient, which is crucial for running large models on limited hardware. Below is a high-level outline of how you could structure this integration:

    Step 1: Install Necessary Libraries

    You'll need to install the Hugging Face transformers library and bitsandbytes for model compression. It's assumed you have whisper and other necessary libraries (from previous steps) already installed.

    pip install transformers bitsandbytes
    

    Step 2: Loading the LLaMA Model with Compression

    from transformers import AutoModelForCausalLM, AutoTokenizer
    import bitsandbytes as bnb
    
    def load_llama_model(model_name="allenai/llama", compression=True):
        tokenizer = AutoTokenizer.from_pretrained(model_name)
        if compression:
            # 8-bit model loading with bitsandbytes for reduced memory footprint
            bnb.optim.GlobalOptimManager.get_instance().set_default_policy('8bit')
        model = AutoModelForCausalLM.from_pretrained(model_name)
        return model, tokenizer
    

    This function loads the LLaMA model with an option for 8-bit quantization. Note that allenai/llama is a placeholder; you'll want to replace it with the actual model identifier from Hugging Face, and ensure it's appropriate for your application (e.g., EleutherAI/llama-13b or similar).

    Step 3: Processing Text to Identify Tasks

    After obtaining the transcribed text from Whisper, you can pass it to LLaMA for further processing to identify tasks or commands. Here's a simple outline for making predictions and parsing them to identify specific tasks:

    def process_text_with_llama(text, model, tokenizer):
        inputs = tokenizer.encode(text, return_tensors="pt")
        
        # Generate a response or prediction
        output_sequences = model.generate(input_ids=inputs, max_length=50)
        
        # Decode the response
        response_text = tokenizer.decode(output_sequences[0], skip_special_tokens=True)
        
        # Process the response to identify the task/command
        task = identify_task(response_text)
        return task
    
    def identify_task(response):
        # This is a placeholder function - you'll need to implement the task identification logic
        # This could be simple keyword matching, or a more sophisticated intent recognition system
        if "write note" in response:
            return "Write Note"
        elif "check news" in response:
            return "Check News"
        elif "weather" in response:
            return "Check Weather"
        # Add more conditions as needed for your application
    
        return "Unknown"
    

    Step 4: Integration with Command Handlers

    Once you've identified the task, you can route the commands to their respective handlers:

    def handle_command(task):
        if task == "Write Note":
            # Call your note-taking function
        elif task == "Check News":
            # Call your news-checking function
        elif task == "Check Weather":
            # Call your weather-checking function
        # Add more handlers as needed
    
    # Example Usage
    model, tokenizer = load_llama_model()
    text_from_speech = "Reminder to check the weather tomorrow for the trip."
    task = process_text_with_llama(text_from_speech, model, tokenizer)
    handle_command(task)
    

    Step 5: Adapt and Expand

    You can expand this system by implementing more sophisticated intent recognition within the identify_task function, or by increasing the variety of commands and tasks your application can understand and execute. For instance, integrating with APIs for checking news, weather, or managing a calendar.

    Additional Notes:

    • Testing and Tuning: It's crucial to test the system thoroughly with various speech inputs to ensure the recognition and task identification work as expected.
    • Optimization: Keep an eye on performance, especially if running on limited hardware. Model quantization helps, but processing still might be resource-intensive.
    • Security and Privacy: Be mindful of privacy and security, particularly if handling personal or sensitive information in tasks like note-taking or calendar management.

    This framework sets the foundation for creating a powerful assistant tool focused on self-development and cognitive load reduction. By continually refining the model's understanding and response generation, you can create a highly personalized and effective tool.