Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning-pro-high· PerplexityHugging Face LMArena Search Arena 24K

Tell me about the technical specifications of the HW used for inference of large neural…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning’s answer.
  1. User

    Tell me about the technical specifications of the HW used for inference of large neural networks (billion of parameters) produced by IBM. Like what chips do they produce that are in production? Please provide benchmark values, like TOPS or TeraFLOPs. Or compare them to one H100 estimating the value of "H100 equivalent".

  2. Perplexity Assistant

    IBM has developed several advanced AI accelerator chips specifically designed for neural network inference, with a focus on energy efficiency and performance.

    NorthPole (AIU NorthPole)

    NorthPole is IBM's cutting-edge AI inference accelerator that was unveiled in 2023. This chip is particularly noteworthy for handling large neural networks with billions of parameters.

    Technical Specifications:

    • Fabricated using a 12nm process
    • Contains 22 billion transistors
    • Die size of 795 square millimeters[6]

    Performance Benchmarks:

    • When running a 3-billion-parameter LLM derived from IBM's Granite-8B-Code-Base model:
      • Achieves latency below 1 millisecond per token
      • 46.9 times faster than the next most energy-efficient GPU
      • 72.7 times more energy efficient than the next lowest latency GPU[6]
    • In a standard 2U server configuration with 16 NorthPole processors connected via PCIe:
      • Achieves throughput of 28,356 tokens per second on the same LLM model[6]
    • For edge applications, NorthPole is 25 times more energy efficient than commonly available 12nm GPUs and 14nm CPUs[6]

    While IBM doesn't publish traditional TOPS or TFLOPS figures for NorthPole in the available information, its performance is measured more in terms of relative efficiency and speed for specific workloads rather than raw compute numbers.

    IBM Analog AI Chip (Hermes)

    IBM also developed an innovative analog in-memory computing chip aimed at deep learning inference tasks, which represents a fundamental architectural shift from traditional digital processors.

    Technical Specifications:

    • Fabricated in 14nm CMOS technology
    • Features 64 analog in-memory compute cores (tiles)
    • Each core contains a 256×256 crossbar array of synaptic unit cells
    • Uses phase-change memory (PCM) devices for weight storage
    • Integrates analog-to-digital converters within each tile
    • Contains lightweight digital processing units for activation functions and scaling operations
    • Includes a global digital processing unit for more complex operations[1][10]

    Performance Benchmarks:

    • Throughput of 400 GOPS/mm² for 8-bit input-output matrix multiplications
    • Over 15 times higher throughput than previous multi-core, in-memory computing chips based on resistive memory
    • Comparable energy efficiency to earlier technologies[1]
    • Achieved 92.81% accuracy on the CIFAR-10 image dataset, which IBM claims is the highest level of accuracy reported for similar technology[1]
    • When benchmarked against other analog in-memory chips (including NeuRRAM, Mythic, and TSMC), it performed matrix-vector multiplications at least 15× faster with comparable energy efficiency[10]

    Comparison to NVIDIA H100

    While there's no direct "H100 equivalent" measure provided for IBM's chips, we can consider the relative positioning:

    The NVIDIA H100, for reference, provides:

    • 67 TFLOPS for FP32 operations
    • 1979 TFLOPS for FP16 Tensor Core operations
    • 3958 TFLOPS for FP8 Tensor Core operations[7]

    IBM's focus with both NorthPole and their analog AI chips has been more on energy efficiency and specific workload performance rather than competing directly on raw FLOPS. The NorthPole chip demonstrates significant advantages in terms of energy efficiency and latency for LLM inference specifically, which may make it more suitable for certain deployment scenarios despite potentially lower raw computational throughput compared to H100.

    IBM Research is continuing to enhance these technologies, with plans to build larger analog in-memory chips that can execute complete network operations end-to-end, potentially rivaling digital accelerators in more comprehensive workloads[10].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.