Recommend me the best edge TPU or other portable solution to run a Whisper +…
- User
Recommend me the best edge TPU or other portable solution to run a Whisper + 1.5b-parameters llm with the lower cost possible. Only provide solution that are in stock or avialable
Assistant
For a cost-effective, portable solution to run Whisper and a 1.5B-parameter LLM, the Radxa Fogwise® AirBox is the top recommendation. Here’s why:
Key Features
- Performance:
- Runs DeepSeek-R1-Distill-Qwen-1.5B at 30 tokens/s with INT4 quantization[3][5].
- Compatible with Whisper for speech-to-text tasks[3][5].
- Hardware:
- 32 TOPS computational power with support for INT8/FP16/BF16/FP32 precision.
- Up to 32GB RAM and expandable storage (supports 4TB via external drives)[3][5].
- Cost Efficiency:
- Open-source software stack and pre-optimized models reduce deployment costs.
- Lower power consumption compared to GPU-based solutions[3][5].
Availability
- Currently in stock and openly documented on Radxa’s website[3][5].
- Fully compatible with edge-optimized frameworks like TensorFlow Lite and PyTorch Mobile.
Alternatives (If AirBox Unavailable)
- Coral TPU (Edge TPU):
- Limited to smaller models (e.g., Whisper-Lite) due to 8MB memory[7]. Not suitable for 1.5B LLMs.
- Raspberry Pi 5 + eGPU:
- Requires external GPUs (e.g., NVIDIA Jetson Orin Nano) for LLM inference, increasing cost and complexity[10].
The Radxa AirBox provides the best balance of portability, performance, and cost for running both Whisper and a 1.5B LLM at the edge.
- Performance:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.