Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview-high-loc· OpenAIHugging Face LMArena Search Arena 24K

Recommend me the best edge TPU or other portable solution to run a Whisper +…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning-pro-high’s answer.
  1. User

    Recommend me the best edge TPU or other portable solution to run a Whisper + 1.5b-parameters llm with the lower cost possible. Only provide solution that are in stock or avialable

  2. Assistant

    Running both Whisper and a 1.5-billion-parameter language model (LLM) on a portable, cost-effective device presents significant challenges due to the substantial computational and memory requirements of these models. Edge devices typically lack the necessary resources to handle such demanding tasks efficiently.

    For instance, the Google Coral Dev Board, equipped with an Edge TPU, excels in running quantized TensorFlow models with low power consumption and fast inference times. However, it may not be suitable for large-scale models like Whisper or a 1.5B-parameter LLM. Similarly, the NVIDIA Jetson Nano offers superior performance for smaller models but may struggle with the memory and processing demands of larger models[1].

    Given these constraints, a more viable solution would be to utilize cloud-based services for running these models. Cloud platforms provide the necessary computational power and scalability to handle large models efficiently. This approach allows you to access high-performance hardware without the need for significant upfront investment in physical devices.

    If portability is a strict requirement, consider using a high-performance laptop equipped with a powerful GPU, such as an NVIDIA RTX series card. While this option involves higher initial costs, it offers the flexibility to run demanding models locally. Ensure that the laptop has sufficient RAM (at least 16GB, preferably 32GB or more) and a high-end GPU to accommodate the resource-intensive nature of these models.

    In summary, while edge devices offer portability and cost-effectiveness, they are generally not suitable for running large models like Whisper and a 1.5B-parameter LLM. Cloud-based solutions or high-performance laptops are more appropriate for these tasks, balancing computational requirements with cost and portability considerations.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.