error : ubuntu@ubuntu-virtual-machine:~/Desktop/new$ ./main.py…
- User
error : ubuntu@ubuntu-virtual-machine:~/Desktop/new$ ./main.py /home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/core/io.py:43: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call. torchaudio.set_audio_backend("soundfile") /home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_verification.py:43: UserWarning: torchaudio._backend.get_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call. backend = torchaudio.get_audio_backend() INFO:speechbrain.utils.quirks:Applied quirks (see
speechbrain.utils.quirks): [allow_tf32, disable_jit_profiling] INFO:speechbrain.utils.quirks:Excluded quirks specified by theSB_DISABLE_QUIRKSenvironment (comma-separated list): [] /home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_verification.py:45: UserWarning: Module 'speechbrain.pretrained' was deprecated, redirecting to 'speechbrain.inference'. Please update your script. This is a change from SpeechBrain 1.0. See: https://github.com/speechbrain/speechbrain/releases/tag/v1.0.0 from speechbrain.pretrained import ( /home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_verification.py:53: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call. torchaudio.set_audio_backend(backend) /home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/tasks/segmentation/mixins.py:37: UserWarning:torchaudio.backend.common.AudioMetaDatahas been moved totorchaudio.AudioMetaData. Please update the import path. from torchaudio.backend.common import AudioMetaData Initializing AudioProcessor with cpu Loading Whisper v2 model: base No language specified, language will be first be detected for each audio file (increases inference time). Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.4.0. To apply the upgrade to your files permanently, runpython -m pytorch_lightning.utilities.upgrade_checkpoint ../../.cache/torch/whisperx-vad-segmentation.binModel was trained with pyannote.audio 0.0.1, yours is 3.1.1. Bad things might happen unless you revert pyannote.audio to 0.x. Model was trained with torch 1.10.0+cu102, yours is 2.5.1. Bad things might happen unless you revert torch to 1.x. Loading Demucs model... Loading VAD model... Downloading: "https://github.com/snakers4/silero-vad/zipball/master" to /home/ubuntu/.cache/torch/hub/master.zip Models loaded successfully! === Extracting Vocals Debug Information === Input audio path: /home/ubuntu/Downloads/audio.mp3- Loading audio file...
- Audio loaded successfully
- Audio shape: torch.Size([2, 599040])
- Sample rate: 48000
- Audio moved to device: cpu
- Chunk duration: 30 seconds
- Chunk samples: 1440000
- Total audio samples: 599040
- Processing full audio...
- Audio shape after unsqueeze: torch.Size([1, 2, 599040])
- apply_model result:
- Type of sources: <class 'torch.Tensor'>
- Sources shape: torch.Size([1, 4, 2, 599040])
- Vocals shape: torch.Size([2, 599040])
Audio shape before transpose: (2, 599040)
Transposed audio shape: (599040, 2)
Audio min: -0.6916637420654297, max: 0.8381609320640564, mean: -0.00017020900850184262
Audio data written to /home/ubuntu/Desktop/new/temp/temp_in.wav
Read test audio from file with shape (599040, 2) and sample rate 48000
ffmpeg stdout:
ffmpeg stderr: ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from '/home/ubuntu/Desktop/new/temp/temp_in.wav':
Duration: 00:00:12.48, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (pcm_s16le (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, wav, to '/home/ubuntu/Desktop/new/temp/temp_out.wav':
Metadata:
ISFT : Lavf58.76.100
Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
Metadata:
encoder : Lavc58.134.100 pcm_s16le
size= 0kB time=00:00:00.00 bitrate=N/A speed=N/A
size= 2304kB time=00:00:09.57 bitrate=1970.3kbits/s speed=17.4x
size= 2340kB time=00:00:12.47 bitrate=1536.1kbits/s speed=22.7x
video:0kB audio:2340kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.003255%
Applying VAD... Adding markers... Applying compression... Transcribing... Transcription error: FasterWhisperPipeline.transcribe() got an unexpected keyword argument 'beam_size' Transcribing... Transcription error: FasterWhisperPipeline.transcribe() got an unexpected keyword argument 'beam_size' Transcribing... Transcription error: FasterWhisperPipeline.transcribe() got an unexpected keyword argument 'beam_size'
script :
import whisperx import torch import torchaudio from demucs.pretrained import get_model from demucs.apply import apply_model import numpy as np import soundfile as sf import subprocess from pathlib import Path import os from config import * from torch.hub import load as torch_hub_load from silero_vad import load_silero_vad, get_speech_timestamps
class AudioProcessor: def init(self, model_size=DEFAULT_MODEL_SIZE, whisper_version=DEFAULT_WHISPER_VERSION): torch.set_num_threads(CPU_THREADS) print(f"Initializing AudioProcessor with {DEVICE}") self.init_models(model_size, whisper_version)
def init_models(self, model_size, whisper_version): try: print(f"Loading Whisper {whisper_version} model: {model_size}") self.whisper_model = whisperx.load_model( model_size, device=DEVICE, compute_type="float32", download_root=str(MODELS_DIR), language=None, # Set to None for automatic language detection, or specify 'en' for English ) print("Loading Demucs model...") self.demucs_model = get_model(name="htdemucs_ft") self.demucs_model.to(DEVICE) print("Loading VAD model...") # Correct way to load Silero VAD self.vad_model, utils = torch_hub_load( repo_or_dir='snakers4/silero-vad', model='silero_vad', force_reload=True ) # Unpack utils properly (self.get_speech_timestamps, self.save_audio, self.read_audio, self.VADIterator, self.collect_chunks) = utils self.vad_model = self.vad_model.to(DEVICE) print("Models loaded successfully!") except Exception as e: print(f"Error initializing models: {e}") raise def process_file(self, input_file, output_file, beam_size=2, patience=1, temperature=0.0): """Main processing pipeline""" try: # 1. Extract vocals vocals, sr = self.extract_vocals(input_file) # 2. Normalize and remove silence normalized = self.normalize_audio(vocals, sr) # 3. Apply VAD vad_audio = self.apply_vad(normalized) # 4. Add markers marked_audio = self.add_markers(vad_audio) # 5. Apply compression compressed = self.apply_compression(marked_audio) # Try transcription with different approaches transcription = None # With markers transcription = self.transcribe(compressed, beam_size, patience, temperature) if self.verify_transcription(transcription): return self.save_transcription(transcription, output_file) # With inverted markers transcription = self.transcribe(-compressed, beam_size, patience, temperature) if self.verify_transcription(transcription): return self.save_transcription(transcription, output_file) # Without markers transcription = self.transcribe(vad_audio, beam_size, patience, temperature) return self.save_transcription(transcription, output_file) except Exception as e: print(f"Error processing file: {e}") raise finally: # Implement cleanup_temp if necessary pass # Replace this with cleanup_temp() if you have that function def extract_vocals(self, audio_path): """Extract vocals using Demucs""" print("=== Extracting Vocals Debug Information ===") print(f"Input audio path: {audio_path}") try: # Load audio print("1. Loading audio file...") audio, sr = torchaudio.load(audio_path) print(f"- Audio loaded successfully") print(f"- Audio shape: {audio.shape}") print(f"- Sample rate: {sr}") # Move to device audio = audio.to(DEVICE) print(f"- Audio moved to device: {DEVICE}") # Chunk processing setup chunk_samples = int(CHUNK_DURATION * sr) print(f"- Chunk duration: {CHUNK_DURATION} seconds") print(f"- Chunk samples: {chunk_samples}") print(f"- Total audio samples: {audio.shape[1]}") # Source list in order sources_list = ['drums', 'bass', 'other', 'vocals'] vocals_index = sources_list.index('vocals') # Should be index 3 # Process chunked or full audio if audio.shape[1] > chunk_samples: print("2. Processing audio in chunks...") vocals_chunks = [] for i in range(0, audio.shape[1], chunk_samples): chunk = audio[:, i:min(i+chunk_samples, audio.shape[1])] print(f" - Processing chunk {i//chunk_samples + 1}") print(f" - Chunk shape before unsqueeze: {chunk.shape}") # Add batch dimension chunk = chunk.unsqueeze(0) # Shape becomes (1, channels, length) print(f" - Chunk shape after unsqueeze: {chunk.shape}") try: # Apply model with torch.no_grad(): sources = apply_model(self.demucs_model, chunk, device=DEVICE) print(" - apply_model result:") print(f" - Type of sources: {type(sources)}") print(f" - Sources shape: {sources.shape}") # Extract vocals if isinstance(sources, torch.Tensor): vocals_chunk = sources[:, vocals_index, :, :] vocals_chunk = vocals_chunk.squeeze(0) else: raise ValueError("Unexpected sources type or structure") if vocals_chunk is None: raise ValueError("No vocals extracted from chunk") print(f" - Vocals chunk shape: {vocals_chunk.shape}") vocals_chunks.append(vocals_chunk) except Exception as chunk_error: print(f" !! Error processing chunk {i//chunk_samples + 1}: {chunk_error}") raise vocals = torch.cat(vocals_chunks, dim=1) print(f"- Concatenated vocals shape: {vocals.shape}") else: print("2. Processing full audio...") # Add batch dimension audio = audio.unsqueeze(0) # Shape becomes (1, channels, length) print(f" - Audio shape after unsqueeze: {audio.shape}") # Apply model with torch.no_grad(): sources = apply_model(self.demucs_model, audio, device=DEVICE) print(" - apply_model result:") print(f" - Type of sources: {type(sources)}") print(f" - Sources shape: {sources.shape}") # Extract vocals if isinstance(sources, torch.Tensor): vocals = sources[:, vocals_index, :, :] vocals = vocals.squeeze(0) else: raise ValueError("Unexpected sources type or structure") if vocals is None: raise ValueError("No vocals extracted from full audio") print(f" - Vocals shape: {vocals.shape}") return vocals, sr except Exception as e: print("=== FATAL ERROR IN EXTRACT_VOCALS ===") print(f"Error details: {e}") import traceback traceback.print_exc() raise def normalize_audio(self, audio, sr): """Normalize audio and remove silence using ffmpeg""" import soundfile as sf temp_in = TEMP_DIR / "temp_in.wav" temp_out = TEMP_DIR / "temp_out.wav" try: # Ensure TEMP_DIR exists TEMP_DIR.mkdir(parents=True, exist_ok=True) # Convert audio tensor to NumPy array audio_np = audio.cpu().numpy() print(f"Audio shape before transpose: {audio_np.shape}") # Transpose if necessary if audio_np.ndim == 2: audio_np = audio_np.T # Shape becomes (samples, channels) print(f"Transposed audio shape: {audio_np.shape}") # Ensure data type is float32 if audio_np.dtype != np.float32: audio_np = audio_np.astype(np.float32) # Check for NaNs or Infs if np.isnan(audio_np).any() or np.isinf(audio_np).any(): raise ValueError("Audio data contains NaNs or Infs.") # Print audio stats print(f"Audio min: {audio_np.min()}, max: {audio_np.max()}, mean: {audio_np.mean()}") # Write audio data to temp_in.wav sf.write(str(temp_in), audio_np, sr, format='WAV', subtype='PCM_16') print(f"Audio data written to {temp_in}") # Test if the file can be read test_audio, test_sr = sf.read(str(temp_in)) print(f"Read test audio from file with shape {test_audio.shape} and sample rate {test_sr}") # Run ffmpeg to normalize and remove silence cmd = [ 'ffmpeg', '-y', '-i', str(temp_in), '-af', ( f"silenceremove=stop_periods=-1:stop_duration=1:stop_threshold={VAD_THRESHOLD}dB," "loudnorm=I=-16:TP=-1.5:LRA=11" ), '-ar', str(sr), str(temp_out) ] process = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) print(f"ffmpeg stdout: {process.stdout}") print(f"ffmpeg stderr: {process.stderr}") if process.returncode != 0: raise Exception(f"ffmpeg error: {process.stderr}") normalized, sr = torchaudio.load(str(temp_out)) normalized = normalized.squeeze(0) return normalized except Exception as e: print(f"Error normalizing audio: {e}") raise def apply_vad(self, audio): """Apply Voice Activity Detection""" print("Applying VAD...") audio_mono = audio.mean(dim=0) if audio.dim() > 1 else audio audio_numpy = audio_mono.cpu().numpy() # Process in chunks chunk_samples = int(CHUNK_DURATION * SAMPLE_RATE) speech_timestamps = [] for i in range(0, len(audio_numpy), chunk_samples): chunk = audio_numpy[i:min(i + chunk_samples, len(audio_numpy))] chunk_tensor = torch.FloatTensor(chunk).to(DEVICE) # Get speech timestamps for the chunk chunk_timestamps = self.get_speech_timestamps( chunk_tensor, self.vad_model, sampling_rate=SAMPLE_RATE ) # Adjust timestamps for chunk position for ts in chunk_timestamps: ts['start'] += i ts['end'] += i speech_timestamps.extend(chunk_timestamps) # Create masked audio masked_audio = torch.zeros_like(audio_mono) for ts in speech_timestamps: start_frame = ts['start'] end_frame = ts['end'] if start_frame < len(masked_audio) and end_frame <= len(masked_audio): masked_audio[start_frame:end_frame] = audio_mono[start_frame:end_frame] return masked_audio def add_markers(self, audio): """Add voice markers at speech boundaries""" print("Adding markers...") marked_audio = audio.clone() # Generate marker signal t = np.arange(int(MARKER_DURATION * SAMPLE_RATE)) marker = torch.from_numpy( 0.1 * np.sin(2 * np.pi * MARKER_FREQUENCY * t / SAMPLE_RATE) ).float() # Find speech segments speech_segments = self.find_speech_segments(audio) # Add markers at segment boundaries for start, end in speech_segments: if start + len(marker) < len(marked_audio): marked_audio[start:start + len(marker)] += marker if end - len(marker) >= 0: marked_audio[end - len(marker):end] += marker return marked_audio def find_speech_segments(self, audio, threshold=0.01): """Find speech segments in audio""" is_speech = torch.abs(audio) > threshold changes = torch.diff(is_speech.long()) change_points = torch.nonzero(changes).flatten() segments = [] for i in range(0, len(change_points)-1, 2): start = change_points[i].item() end = change_points[i+1].item() if i+1 < len(change_points) else len(audio)-1 segments.append((start, end)) return segments def apply_compression(self, audio): """Apply speech compression using ffmpeg""" print("Applying compression...") temp_in = TEMP_DIR / "temp_compress_in.wav" temp_out = TEMP_DIR / "temp_compress_out.wav" try: sf.write(str(temp_in), audio.cpu().numpy(), SAMPLE_RATE) cmd = [ 'ffmpeg', '-y', '-i', str(temp_in), '-af', 'acompressor=threshold=-12dB:ratio=2:attack=200:release=1000', '-ar', str(SAMPLE_RATE), str(temp_out) ] subprocess.run(cmd, check=True, capture_output=True) compressed, _ = torchaudio.load(str(temp_out)) return compressed.squeeze() except Exception as e: print(f"Error applying compression: {e}") raise def transcribe(self, audio, beam_size=2, patience=1, temperature=0.0): """Transcribe audio using WhisperX""" print("Transcribing...") temp_file = TEMP_DIR / "temp_transcribe.wav" try: sf.write(str(temp_file), audio.cpu().numpy(), SAMPLE_RATE) # Load and transcribe audio_data = whisperx.load_audio(str(temp_file)) result = self.whisper_model.transcribe( audio_data, batch_size=1, # Optimized for CPU beam_size=beam_size, patience=patience, temperature=temperature, ) # Align timestamps model_a, metadata = whisperx.load_align_model( language_code=result["language"], device=DEVICE ) result = whisperx.align( result["segments"], model_a, metadata, audio_data, DEVICE, extend_duration=0.0 ) return result except Exception as e: print(f"Transcription error: {e}") return None def verify_transcription(self, transcription): """Verify if transcription contains markers""" if not transcription or 'segments' not in transcription: return False marker_patterns = ["*", "beep", "tone", "[noise]", "[sound]"] return any( any(pattern in segment['text'].lower() for pattern in marker_patterns) for segment in transcription['segments'] ) def save_transcription(self, transcription, output_file): """Save transcription to file""" if not transcription: return False try: with open(output_file, 'w', encoding='utf-8') as f: for segment in transcription['segments']: f.write( f"[{segment['start']:.2f} - {segment['end']:.2f}] {segment['text']}\n" ) return True except Exception as e: print(f"Error saving transcription: {e}") return False - Assistant
The error you're encountering occurs because the
transcribemethod you're using does not acceptbeam_size,patience, ortemperatureas direct keyword arguments. Instead, these parameters should be provided within adecoding_optionsdictionary when calling thetranscribemethod.Here's how you can modify your
transcribemethod to fix the error:Original Code:
result = self.whisper_model.transcribe( audio_data, batch_size=1, # Optimized for CPU beam_size=beam_size, patience=patience, temperature=temperature, )Modified Code:
decoding_options = { "beam_size": beam_size, "patience": patience, "temperature": temperature, } result = self.whisper_model.transcribe( audio_data, batch_size=1, # Optimized for CPU decoding_options=decoding_options, )Explanation:
-
Issue: The
transcribemethod of theFasterWhisperPipeline(which is used internally bywhisperx.load_model) does not acceptbeam_size,patience, ortemperatureas direct keyword arguments. These must be passed within adecoding_optionsdictionary. -
Solution: Create a
decoding_optionsdictionary containing your decoding parameters (beam_size,patience,temperature) and pass it to thetranscribemethod using thedecoding_optionskeyword argument.
Here's the updated
transcribemethod in your script:def transcribe(self, audio, beam_size=2, patience=1, temperature=0.0): """Transcribe audio using WhisperX""" print("Transcribing...") temp_file = TEMP_DIR / "temp_transcribe.wav" try: sf.write(str(temp_file), audio.cpu().numpy(), SAMPLE_RATE) # Load and transcribe audio_data = whisperx.load_audio(str(temp_file)) # Prepare decoding options decoding_options = { "beam_size": beam_size, "patience": patience, "temperature": temperature, } result = self.whisper_model.transcribe( audio_data, batch_size=1, # Optimized for CPU decoding_options=decoding_options, ) # Align timestamps model_a, metadata = whisperx.load_align_model( language_code=result["language"], device=DEVICE ) result = whisperx.align( result["segments"], model_a, metadata, audio_data, DEVICE, extend_duration=0.0 ) return result except Exception as e: print(f"Transcription error: {e}") return NoneAdditional Notes:
-
Documentation Reference: This change aligns with the API of the
FasterWhisperPipelinein thewhisperxlibrary, where decoding parameters are expected to be passed via thedecoding_optionsargument. -
Other Considerations:
- Ensure that your versions of
whisperx,faster-whisper, and related dependencies are up-to-date and compatible. - Since you're using several third-party libraries that may have frequent updates, always refer to their latest documentation when encountering such errors.
- Ensure that your versions of
By modifying your code as shown, the
transcribemethod should now correctly receive the decoding parameters, and the error should be resolved.
Summary:
- Error Cause: Passing decoding parameters (
beam_size,patience,temperature) directly to thetranscribemethod, which doesn't accept them as direct keyword arguments. - Solution: Pass these parameters within a
decoding_optionsdictionary to thetranscribemethod using thedecoding_optionskeyword argument.
-