Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-search-preview· OpenAIHugging Face LMArena Search Arena 24K

How do i get vLLM RUnning on my M2 Macs GPU instead of the CPU ?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-pro’s answer.
  1. User

    How do i get vLLM RUnning on my M2 Macs GPU instead of the CPU ?

  2. Assistant

    As of March 2025, vLLM does not natively support GPU acceleration on macOS, including M2 Macs. The current implementation is limited to CPU usage, with the VLLM_TARGET_DEVICE automatically set to cpu on macOS systems[1].

    There have been discussions within the vLLM community regarding the integration of GPU support via Apple's Metal API. However, these efforts are still in the experimental phase, and no official release has been made[2].

    For GPU-accelerated machine learning tasks on macOS, you might consider using frameworks that support Apple's Metal API, such as TensorFlow with the tensorflow-metal plugin or PyTorch with Metal support. These frameworks are designed to leverage the GPU capabilities of M1 and M2 Macs, potentially offering performance improvements for your models[3].

    In summary, while vLLM currently operates solely on the CPU for macOS users, exploring alternative frameworks with GPU support could provide the acceleration you're seeking.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt