AI benchmarks are broken. Here’s what we need instead.

For many years, artificial intelligence has been assessed by comparing its performance against human capabilities in specific tasks. This evaluation method has been applied across various domains, including games like chess, complex mathematical problems, software development, and even creative writing. The approach of pitting AI directly against human performance on isolated tasks has been a long-standing and appealing way to measure AI progress.

This benchmark approach is being questioned as potentially inadequate for understanding the true capabilities and limitations of modern AI systems.