MLCommons unveils groundbreaking AI benchmark tests for speed and efficiency

MLCommons, a prominent Artificial Intelligence (AI) benchmarking group, has unveiled a fresh suite of benchmarks aimed at evaluating the speed and efficiency of AI applications. These tests focus on measuring how swiftly high-end hardware can execute AI tasks and generate responses, shedding light on the capabilities of AI models like ChatGPT in delivering text-to-image outputs.

The latest benchmarks introduced by MLCommons assess the rapidity at which AI chips and systems can process data and generate responses from sophisticated AI models. Specifically, they provide insights into the speed with which AI applications, such as ChatGPT, can produce responses to user queries, reflecting the performance of these systems in real-world scenarios.

Among the newly introduced benchmarks is Llama 2, designed to evaluate the speed of question-and-answer scenarios for large language models. Developed by Meta 

Platforms, this model boasts an impressive 70 billion parameters, showcasing the cutting-edge capabilities of modern AI architectures. Additionally, MLCommons has expanded its benchmarking toolkit to include a second text-to-image generator, dubbed MLPerf, based on Stability AI’s Stable Diffusion XL model.

In terms of hardware performance, servers equipped with Nvidia’s H100 chips emerged as clear winners across both benchmarks, showcasing superior raw performance. 

Notably, these servers were supplied by leading companies such as Alphabet’s Google, Supermicro, and Nvidia itself. While Nvidia’s L40S chip also featured in submissions from various server builders, a notable entry came from server builder Krai, which leveraged a Qualcomm AI chip for the image generation benchmark, offering a compelling alternative with reduced power consumption.

Intel, a key player in the AI hardware landscape, submitted a design based on its Gaudi2 accelerator chips, garnering praise for its solid performance. However, beyond raw performance, energy efficiency remains a critical consideration in AI deployment. Given the substantial power consumption associated with advanced AI chips, optimising the performance-to-energy ratio poses a significant challenge for AI companies.

MLCommons recognises the importance of energy efficiency and has established a separate benchmark category dedicated to measuring power consumption, providing comprehensive insights into the performance and sustainability of AI systems.