what options do i have for in client side browser llms
- User
what options do i have for in client side browser llms
Assistant
There are several options for running LLMs directly in the browser:
WebLLM is probably the most mature option. It uses WebGPU for acceleration and can run models like Llama 2, Vicuna, and others. It's developed by the MLC team and offers good performance for smaller models.
Transformers.js by Hugging Face lets you run ONNX-converted models in the browser using WebAssembly. It supports a wide range of models but performance varies significantly based on model size.
ONNX.js provides a more general framework for running ONNX models in browsers, which includes many language models that have been converted to the ONNX format.
MediaPipe from Google has some text generation capabilities, though it's more focused on multimodal applications.
Web-based implementations of smaller models like GPT-2 or DistilBERT are available through various JavaScript libraries.
The main constraints you'll face are memory limitations (most browsers cap WebAssembly memory), computational speed (even with WebGPU, it's much slower than dedicated hardware), and model size (you're generally limited to smaller models under 1-2GB).
WebLLM is probably your best starting point if you want something that works well out of the box. The performance will depend heavily on the user's hardware - newer devices with good GPUs will handle it much better than older or mobile devices.
What's your intended use case? That might help narrow down the best option for your specific needs.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.