AI Benchmarks Miss 95% of Human Tasks
A new analysis has revealed that benchmarks, a type of "exam" for measuring artificial intelligence (AI) performance, fail to effectively evaluate approximately 95% of the detailed tasks that constitute real-world human work. While AI performance continues to improve, the lack of evaluation for how AI performs in actual human tasks could lead to errors in policy decisions related to AI and job market changes, experts warn.
AI training data company **Mucore** unveiled an "interactive map" on the 15th (local time), showing the scope of AI benchmark applications across U.S. occupations. The map applied 29 self-selected benchmarks to 923 occupations and 18,797 related tasks under the U.S. Department of Labor’s occupational classification system. It illustrates whether AI’s ability to perform specific tasks is properly assessed by existing benchmarks.
According to the map, only 208 out of 923 total occupations (22.5%) were adequately evaluated. Among the 18,797 detailed tasks, only 924 (approximately 5%) were properly assessed by benchmarks with five or more questions.
**Mucore** stated, "90% of American working hours are spent on tasks that are rarely or never addressed by existing benchmarks," adding, "It is extremely difficult to determine how well AI models perform the actual work Americans do." Current benchmarks, they argue, cannot reliably predict whether AI can replace human labor.
The application of benchmarks varied significantly across occupational groups. In computer and mathematics fields, 29 out of 36 occupations were measured. In business and finance, 33 out of 48 occupations were assessed, and in legal fields, 5 out of 7 occupations were evaluated. Conversely, only 12 out of 89 medical professions and 9 out of 62 education and library-related occupations were benchmarked.
Benchmarks were predominantly applied to sectors where AI could be easily introduced and generate immediate profit. For example, while lawyer tasks were measured, similar AI-exposed roles like legal secretaries or court clerks were largely unassessed. **Mucore** noted, "Mid-income knowledge-based jobs with smaller scales appear insufficiently measured." Structured office tasks, which AI could quickly replace, lack such benchmark data.
Additionally, many work processes involve physical labor that current benchmarks cannot measure. According to **Mucore**, 63% of U.S. jobs include physical tasks not captured by existing benchmarks.
**Mucore** warned, "If this measurement gap is not addressed, significant blind spots will persist in understanding AI capabilities." A source from the tech industry added, "If we cannot accurately assess whether AI can replace numerous jobs, major disruptions could occur in labor markets and policy domains."
