How to use openai-whisper python package for mic recording recognition on the fly? My…
- User
How to use openai-whisper python package for mic recording recognition on the fly? My goal is to automatically start script on boot (Ubuntu), check availability of input-output audio devices and start recording with stream through ASR (later i want to add commands recognition and "skills", but for now let's just write transcription with timestamps into text file). Basic example from GitHub: ```python import whisper
model = whisper.load_model("base") result = model.transcribe("audio.mp3") print(result["text"])```
- Assistant
Creating a script to recognize speech from the microphone in real-time using OpenAI's Whisper and then automatically running it on boot in an Ubuntu system involves several steps. Below, I'll guide you through setting up a basic example that captures audio from the microphone, transcribes it using Whisper, and writes the transcription with timestamps into a text file. Additionally, I'll show you how to ensure it runs at startup.
Step 1: Install Necessary Packages
First, ensure you have Python 3.6 or later installed. Then, install OpenAI's Whisper and other necessary packages. You might also need
sounddeviceto capture audio from the microphone in real-time andnumpyfor handling audio data.pip install whisper sounddevice numpyStep 2: Create the Python Script
Create a Python script,
whisper_mic.py, to capture audio from the microphone and transcribe it in real-time. This will be a simplified example; you might need to refine it based on your requirements.import numpy as np import sounddevice as sd import whisper import time import json # Load Whisper model model = whisper.load_model("base") # Define callback function to stream audio from mic def callback(indata, frames, time, status): # Check if there's an error capturing audio if status: print(status, file=sys.stderr) # Convert audio data to proper format audio_numpy = np.array(indata[:,0], dtype=np.float32) result = model.transcribe(audio_numpy, verbose=True) print(result["text"]) # Optionally, save the transcription with timestamps into a text file with open("transcription.txt", "a") as file: timestamp = time.ctime() transcription = result["text"] file.write(f"{timestamp}: {transcription}\n") # Start streaming from microphone with sd.InputStream(callback=callback, dtype='float32', channels=1, samplerate=16000): print("Recording...") while True: # Keep the script alive time.sleep(0.1)Step 3: Test the Script
Before setting it to start at boot, manually test the script to ensure it works as expected:
python whisper_mic.pyStep 4: Running the Script on Boot
To run the script at startup, you can use
systemdin Ubuntu.- Create a systemd service file for your script:
sudo nano /etc/systemd/system/whisper_mic.service- Add the following content to the service file:
[Unit] Description=Whisper Microphone Transcription After=multi-user.target [Service] Type=idle User=your_username ExecStart=/usr/bin/python3 /path/to/whisper_mic.py Restart=always [Install] WantedBy=multi-user.targetReplace
your_usernamewith your actual username and/path/to/whisper_mic.pywith the full path to your script.- Enable and start the service:
sudo systemctl daemon-reload sudo systemctl enable whisper_mic.service sudo systemctl start whisper_mic.service- Check the status of your service:
sudo systemctl status whisper_mic.serviceThis is a basic framework. Depending on your exact requirements, such as handling more advanced audio processing, integrating command recognition, or developing specific "skills," you will need to build upon this foundation. Additionally, always monitor the performance and adjust the model or resources as necessary, especially when running heavy workloads like real-time audio processing and transcription on boot-up.
- User
I might want to add noise reduction. Noise profile is more or less constant (mostly from working desktop itself - coolers can be heard). Not sure if we should discard it entirely (noise level can be used to estimate PC load), because anomaly detection is also considered.
- Assistant
Adding noise reduction to your real-time transcription system can significantly improve speech recognition accuracy, especially in environments with constant background noise, such as cooler fans from a desktop. The Whisper model is quite robust in handling various noise conditions, but pre-processing the audio to reduce noise can still be beneficial.
Given your interest in potentially leveraging noise levels for anomaly detection or estimating PC load, it might be useful to take a two-pronged approach: perform noise reduction for the purpose of improving speech recognition, while also analyzing the noise level to extract useful information.
Step 1: Noise Reduction
To perform noise reduction, you can use the
noisereducelibrary, which is a Python package that can automatically reduce noise from audio.- Install the library:
pip install noisereduce- Modify the callback function to incorporate noise reduction:
First, record a short segment of your background noise without speech and save it as a
.wavfile. You'll use this as your noise profile.Modify your
callbackfunction in thewhisper_mic.pyscript to usenoisereduce:import noisereduce as nr # Load your noise profile noise_clip = np.loadtxt('noise_profile.npy') def callback(indata, frames, time, status): if status: print(status, file=sys.stderr) # Convert to numpy array and reduce noise audio_numpy = np.array(indata[:,0], dtype=np.float32) reduced_noise_audio = nr.reduce_noise(y=audio_numpy, sr=16000, y_noise=noise_clip, use_tensorflow=False) result = model.transcribe(reduced_noise_audio, verbose=True) print(result["text"]) with open("transcription.txt", "a") as file: timestamp = time.ctime() transcription = result["text"] file.write(f"{timestamp}: {transcription}\n")Ensure you have created a
noise_profile.npyby recording the background noise and saving the NumPy array usingnp.savetxt('noise_profile.npy', noise_data).Step 2: Noise Level Analysis
If you're interested in using the noise level for estimating PC load or anomaly detection, you can calculate the sound level from the raw audio data and perform analysis or log these levels for review.
- Calculate the sound level:
Add a function to calculate the sound level (dB) from the audio:
def calculate_sound_level(audio): # Convert audio to dB sound_level = 20 * np.log10(np.std(audio)) return sound_level- Integrate the sound level calculation:
Integrate this function into your workflow, possibly before noise reduction, to analyze or log the background noise level for further use in anomaly detection or PC load estimation.
... audio_numpy = np.array(indata[:,0], dtype=np.float32) sound_level = calculate_sound_level(audio_numpy) # You can log or analyze this sound_level as needed ...Considerations
- Real-time Processing: All modifications should ensure that the real-time processing demands do not exceed your system's capabilities. Noise reduction and additional analysis can be computationally expensive.
- Anomaly Detection: If you're interested in anomaly detection, consider establishing a baseline sound level during typical operating conditions. Significant deviations from this baseline could be used to trigger alerts or further investigation.
- Adaptive Noise Cancellation: For more advanced scenarios, consider looking into adaptive noise cancellation techniques that continuously learn and adjust to the background noise in real-time.
This approach allows you to not only improve the accuracy of your speech recognition by reducing noise but also gives you the flexibility to analyze the environment's acoustic characteristics for other applications.
- User
After ASR I want to add LLM (e.g. LLaMA, from huggingface transformers with bitsandbytes 8bit) to process text and identify task. E.g. write notes, check news/weather, calendar integration, etc (focus on self development and reducing cognitive load).
- Assistant
Integrating a Large Language Model (LLM) like LLaMA (from Hugging Face Transformers) into your speech recognition system to process text and identify tasks is an excellent way to build a sophisticated, personal assistant-like application. Utilizing
bitsandbytesfor 8-bit quantization can help make the model more memory efficient, which is crucial for running large models on limited hardware. Below is a high-level outline of how you could structure this integration:Step 1: Install Necessary Libraries
You'll need to install the Hugging Face
transformerslibrary andbitsandbytesfor model compression. It's assumed you havewhisperand other necessary libraries (from previous steps) already installed.pip install transformers bitsandbytesStep 2: Loading the LLaMA Model with Compression
from transformers import AutoModelForCausalLM, AutoTokenizer import bitsandbytes as bnb def load_llama_model(model_name="allenai/llama", compression=True): tokenizer = AutoTokenizer.from_pretrained(model_name) if compression: # 8-bit model loading with bitsandbytes for reduced memory footprint bnb.optim.GlobalOptimManager.get_instance().set_default_policy('8bit') model = AutoModelForCausalLM.from_pretrained(model_name) return model, tokenizerThis function loads the LLaMA model with an option for 8-bit quantization. Note that
allenai/llamais a placeholder; you'll want to replace it with the actual model identifier from Hugging Face, and ensure it's appropriate for your application (e.g.,EleutherAI/llama-13bor similar).Step 3: Processing Text to Identify Tasks
After obtaining the transcribed text from Whisper, you can pass it to LLaMA for further processing to identify tasks or commands. Here's a simple outline for making predictions and parsing them to identify specific tasks:
def process_text_with_llama(text, model, tokenizer): inputs = tokenizer.encode(text, return_tensors="pt") # Generate a response or prediction output_sequences = model.generate(input_ids=inputs, max_length=50) # Decode the response response_text = tokenizer.decode(output_sequences[0], skip_special_tokens=True) # Process the response to identify the task/command task = identify_task(response_text) return task def identify_task(response): # This is a placeholder function - you'll need to implement the task identification logic # This could be simple keyword matching, or a more sophisticated intent recognition system if "write note" in response: return "Write Note" elif "check news" in response: return "Check News" elif "weather" in response: return "Check Weather" # Add more conditions as needed for your application return "Unknown"Step 4: Integration with Command Handlers
Once you've identified the task, you can route the commands to their respective handlers:
def handle_command(task): if task == "Write Note": # Call your note-taking function elif task == "Check News": # Call your news-checking function elif task == "Check Weather": # Call your weather-checking function # Add more handlers as needed # Example Usage model, tokenizer = load_llama_model() text_from_speech = "Reminder to check the weather tomorrow for the trip." task = process_text_with_llama(text_from_speech, model, tokenizer) handle_command(task)Step 5: Adapt and Expand
You can expand this system by implementing more sophisticated intent recognition within the
identify_taskfunction, or by increasing the variety of commands and tasks your application can understand and execute. For instance, integrating with APIs for checking news, weather, or managing a calendar.Additional Notes:
- Testing and Tuning: It's crucial to test the system thoroughly with various speech inputs to ensure the recognition and task identification work as expected.
- Optimization: Keep an eye on performance, especially if running on limited hardware. Model quantization helps, but processing still might be resource-intensive.
- Security and Privacy: Be mindful of privacy and security, particularly if handling personal or sensitive information in tasks like note-taking or calendar management.
This framework sets the foundation for creating a powerful assistant tool focused on self-development and cognitive load reduction. By continually refining the model's understanding and response generation, you can create a highly personalized and effective tool.