Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-search-preview· OpenAIHugging Face LMArena Search Arena 24K

Tell me about the technical specifications of the HW used for inference of large neural…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning-pro-high’s answer.
  1. User

    Tell me about the technical specifications of the HW used for inference of large neural networks (billion of parameters) produced by IBM. Like what chips do they produce that are in production? Please provide benchmark values, like TOPS or TeraFLOPs. Or provide the nearest GPUs (or neural network engine) with similar performance that an end user could buy.

  2. Assistant

    IBM has developed several hardware solutions optimized for the inference of large neural networks, including the Power10 processor, the Telum processor, and the NorthPole inference chip.

    Power10 Processor

    The IBM Power10 processor, introduced in 2021, is designed to enhance AI workloads, particularly inference tasks. Key features include:

    • Core Configuration: Up to 15 cores per chip, each supporting eight-way simultaneous multithreading (SMT8).
    • Clock Speed: Operating at frequencies between 3.5 GHz and 4.0 GHz.
    • Cache: Each core has 48 KB of instruction cache and 32 KB of data cache, with a shared L3 cache of 120 MB per chip.
    • Matrix Math Assist (MMA) Engines: Four MMA engines per core, designed to accelerate matrix multiplication operations common in AI inference.
    • Memory Support: Supports up to 16 TB of DDR4 memory with a bandwidth of 410 GB/s.
    • I/O Capabilities: Equipped with PCIe 5.0 interfaces, offering high-speed connectivity for accelerators and other peripherals.

    While specific TOPS (Tera Operations Per Second) or TeraFLOPS (Tera Floating Point Operations Per Second) benchmarks for the Power10 processor are not publicly disclosed, its architecture is optimized for AI inference tasks, providing a balance between computational power and energy efficiency[1].

    Telum Processor

    The IBM Telum processor, launched in 2021, is tailored for enterprise-class workloads, including AI inference. Notable aspects include:

    • Core Configuration: Comprises eight cores per chip, each with a deep superscalar out-of-order pipeline.
    • Clock Speed: Operating at frequencies exceeding 5 GHz.
    • Cache: Each core is equipped with 32 MB of L2 cache.
    • Neural Network Processing Assists (NNPA): Introduces a new 16-bit floating-point format and specialized instructions to accelerate neural network computations.
    • On-Chip AI Acceleration: Integrates AI inference capabilities directly into the processor, reducing latency and improving throughput for AI applications.

    Specific performance metrics such as TOPS or TeraFLOPS for the Telum processor are not publicly available. However, its design emphasizes high-frequency operation and specialized instructions to enhance AI inference performance[2].

    NorthPole Inference Chip

    IBM's NorthPole inference chip, presented in 2024, is engineered for high-performance AI inference. Key specifications include:

    • Core Array: Features a 256-core array.
    • Memory: Contains 192 MB of distributed SRAM.
    • Clock Speed: Operates at a nominal frequency of 400 MHz.
    • Performance: Achieves over 200 TOPS at 8-bit precision, 400 TOPS at 4-bit precision, and 800 TOPS at 2-bit precision.
    • Process Technology: Manufactured using a 12 nm process, with a die size of 795 mm² and 22 billion transistors.

    The NorthPole chip is designed to deliver high throughput and energy efficiency for large-scale AI inference tasks[3].

    Comparison with Commercial GPUs

    For end-users seeking hardware with similar performance characteristics, GPUs from NVIDIA and AMD are prominent options:

    • NVIDIA A100 Tensor Core GPU: Delivers up to 312 TFLOPS of AI performance, supporting both training and inference workloads.
    • AMD Instinct MI300A: Provides up to 980.6 TFLOPS of FP16 performance, with 5.3 TB/s of memory bandwidth, suitable for AI inference tasks[4].

    While IBM's processors like Power10 and Telum are optimized for enterprise environments and may not be directly available for individual purchase, GPUs such as the NVIDIA A100 and AMD Instinct MI300A offer comparable performance for large-scale AI inference applications.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.