Skip to content
TheChatArchiveThe Chat Archive
Metallama-4-scout-17b-16e-instruct· MetaHugging Face LMArena Human Preference 140K

What sized llm models can you run inference well on an M4 Max macbook pro? I know memory…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-30b-a3b’s answer.
  1. User

    What sized llm models can you run inference well on an M4 Max macbook pro? I know memory bandwidth is usually the limit, so the full sized "max" seems best here. I'm wondering if 128GB memory actually matters over 64GB if models that fit in 64GB already start to get too slow

  2. Meta Assistant

    The M4 Max MacBook Pro is a powerful machine, and running large language models (LLMs) on it can be a great way to leverage its capabilities.

    To estimate which sized LLM models can run inference well on an M4 Max MacBook Pro, let's consider the memory bandwidth and capacity.

    Memory Bandwidth: The M4 Max MacBook Pro has a memory bandwidth of up to 200 GB/s ( Apple claims 200 GB/s for the M4 Max with 128GB RAM). This is a crucial factor in determining the model's performance.

    Memory Capacity: You have two options: 64GB and 128GB. We'll discuss how this affects model size and performance.

    Model Size and Inference Speed: The size of the model is typically measured in terms of the number of parameters (e.g., 7B, 13B, 30B, etc.). The model's memory footprint is roughly proportional to the number of parameters.

    As a rough estimate, here are some popular LLM models and their memory footprints:

    • 7B models (e.g., LLaMA-7B): around 14-16 GB of memory
    • 13B models (e.g., LLaMA-13B): around 28-32 GB of memory
    • 30B models (e.g., LLaMA-30B): around 60-70 GB of memory

    Assuming a model with a similar architecture and a moderate activation size, here are some rough estimates of the maximum model sizes that can be run on an M4 Max MacBook Pro with 64GB and 128GB of RAM:

    • 64GB RAM:
      • Small to medium models: 7B-13B parameters (~14-32 GB of memory)
      • Large models: up to 20B-25B parameters (~40-50 GB of memory), but with potential slowdowns due to memory bandwidth limitations
    • 128GB RAM:
      • Medium to large models: 13B-30B parameters (~28-70 GB of memory)
      • Extra-large models: up to 40B-50B parameters (~80-100 GB of memory), with more comfortable headroom for memory bandwidth

    Keep in mind that these estimates are rough and can vary depending on the specific model architecture, framework, and other factors.

    Do 128GB matter over 64GB? If models that fit in 64GB already start to get too slow, the answer is: it depends.

    If you're working with smaller models (e.g., 7B-13B), the 64GB option might be sufficient. However, if you're targeting larger models (e.g., 20B-30B), the 128GB option will provide more comfortable headroom for memory bandwidth and allow for smoother inference.

    In general, if you plan to work with larger models or want to future-proof your setup, the 128GB option is likely a better choice. However, if you're on a budget or primarily work with smaller models, the 64GB option might still be sufficient.

    Inference Optimization To maximize inference performance on your M4 Max MacBook Pro, consider the following:

    1. Use optimized frameworks: Leverage frameworks like Hugging Face's Transformers, TensorFlow, or PyTorch, which have optimized implementations for Apple Silicon.
    2. Quantization: Apply quantization techniques (e.g., int8, int16) to reduce the model's memory footprint and computational requirements.
    3. Model pruning: Prune your model to reduce its size and computational requirements.
    4. Batching and parallelization: Use batching and parallelization techniques to maximize memory bandwidth utilization.

    By considering these factors and optimizing your inference setup, you can unlock the full potential of your M4 Max MacBook Pro and run larger, more complex LLM models efficiently.

  3. User

    The full spec M4 Max has a memory bandwidth of 546GB/s, according to apple

  4. Meta Assistant

    The full-spec M4 Max MacBook Pro actually has a memory bandwidth of 546 GB/s, not 200 GB/s. This is a significant increase and will have a substantial impact on the performance of large language models.

    With this updated information, let's reassess the estimates:

    Model Size and Inference Speed: Assuming a model with a similar architecture and a moderate activation size, here are some revised estimates of the maximum model sizes that can be run on an M4 Max MacBook Pro with 64GB and 128GB of RAM:

    • 64GB RAM:
      • Small to medium models: 30B-40B parameters (~60-80 GB of memory)
      • Large models: up to 50B-60B parameters (~100-120 GB of memory), with some potential slowdowns due to memory bandwidth limitations
    • 128GB RAM:
      • Medium to large models: 50B-70B parameters (~100-140 GB of memory)
      • Extra-large models: up to 100B-120B parameters (~200-240 GB of memory), with more comfortable headroom for memory bandwidth

    Keep in mind that these estimates are still rough and can vary depending on the specific model architecture, framework, and other factors.

    Do 128GB matter over 64GB? With the increased memory bandwidth, the 128GB option provides a more significant advantage. If you're working with larger models (e.g., 50B-70B), the 128GB option will provide more comfortable headroom for memory bandwidth and allow for smoother inference.

    In general, if you plan to work with larger models or want to future-proof your setup, the 128GB option is likely a better choice. However, if you're on a budget or primarily work with smaller models, the 64GB option might still be sufficient.

    Inference Optimization To maximize inference performance on your M4 Max MacBook Pro, consider the same optimizations as before:

    1. Use optimized frameworks: Leverage frameworks like Hugging Face's Transformers, TensorFlow, or PyTorch, which have optimized implementations for Apple Silicon.
    2. Quantization: Apply quantization techniques (e.g., int8, int16) to reduce the model's memory footprint and computational requirements.
    3. Model pruning: Prune your model to reduce its size and computational requirements.
    4. Batching and parallelization: Use batching and parallelization techniques to maximize memory bandwidth utilization.

    With the M4 Max's impressive memory bandwidth and your optimized setup, you'll be able to run larger and more complex LLM models efficiently!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.