Skip to content
TheChatArchiveThe Chat Archive
Mistralmagistral-medium-2506· Mistral AIHugging Face LMArena Human Preference 140K

ok how to speed up, use c++ whispler version ? how faster it will be, any other…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-235b-a22b-no-thinking’s answer.
  1. User

    ok how to speed up, use c++ whispler version ? how faster it will be, any other optimialization?

  2. Mistral Assistant

    To speed up your real-time translation system, the most effective optimization is to switch to FasterWhisper, which is a reimplementation of Whisper in C++ with significant speed improvements. Here's how to implement it:

    1. Install FasterWhisper:
    pip install faster-whisper
    
    1. Modify your code to use FasterWhisper and pre-load translation models:
    from faster_whisper import WhisperModel
    
    def load_whisper_model(model_name="base"):
        print("Loading Whisper model...")
        device = "cuda" if torch.cuda.is_available() else "cpu"
        print(f"Using device: {device}")
        model = WhisperModel(model_name, device=device, compute_type="float16" if device == "cuda" else "int8")
        return model
    
    def process_audio(audio_data):
        if len(audio_data) < RATE * 0.5:
            return ""
        audio_data = audio_data / np.max(np.abs(audio_data) + 1e-8)
        audio_data = audio_data.astype(np.float32)
        segments, _ = whisper_model.transcribe(audio_data, language="he", vad_filter=True)
        text = " ".join(segment.text for segment in segments)
        return text.strip()
    
    def init_translation_models():
        from_code = "he"
        to_code = "pl"
        try:
            argostranslate.package.update_package_index()
            available_packages = argostranslate.package.get_available_packages()
            he_en_package = next(filter(lambda x: x.from_code == from_code and x.to_code == "en", available_packages), None)
            if he_en_package:
                argostranslate.package.install_from_path(he_en_package.download())
            en_pl_package = next(filter(lambda x: x.from_code == "en" and x.to_code == to_code, available_packages), None)
            if en_pl_package:
                argostranslate.package.install_from_path(en_pl_package.download())
            print("Translation models loaded")
        except Exception as e:
            print(f"Error loading translation models: {e}")
    
    # Call init_translation_models() at startup after loading Whisper model
    

    This implementation should provide a 2-4x speedup in transcription time compared to the original Whisper implementation. Combined with pre-loading translation models, your system should feel much more responsive in real-time use.

    For further speed improvements, consider:

    1. Using a smaller Whisper model if acceptable accuracy can be achieved
    2. Implementing model quantization
    3. Moving more components to C++ (e.g., using Whisper.cpp) for additional speed gains, though this would require more significant code changes.

    The expected performance improvement from these changes should make your real-time translation system much faster and more responsive.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.