When Rare Situations Break AI: Researchers Redefine Corner Cases in Machine
Machine learning models are used in highly automated cars, medical systems, and aviation. How well they work depends on the data they learn from and how they are built. When a model fails in a rare, new, or dangerous situation, researchers call it a corner case. People often blame these failures on training data that is too small or does not cover enough situations. But a new open-access study argues that this is only part of the problem. The model itself and the way it is built also affect when and how failures happen. Prediction errors caused by the model should therefore not come as a surprise.
The study's main contribution is a general definition of a corner case that focuses on the model itself. It also provides a mathematical formula that makes corner cases measurable instead of treating them as isolated examples. The authors point out that machine learning researchers do not share one widely accepted definition of corner cases. The meaning can vary greatly depending on the field where the technology is used.
This problem is especially serious in the automotive industry. Popular datasets such as BDD100K, nuScenes, and Waymo Open were collected in the United States. KITTI and A2D2 come from Germany, while ApolloScape comes from China. Mapillary covers a wider range of places around the world, but its coverage is still uneven. Rainy weather, bright daylight, and nighttime scenes appear more often in datasets, while winter conditions are much less common. BDD100K contains relatively few winter scenes, and the Canadian Adverse Driving Conditions dataset focuses on winter but has other weaknesses. No single dataset can capture every real-world situation. Local factors, such as traffic laws, driving habits, and differences in road signs and symbols, also make it harder for a dataset to represent every possible situation.
When a trained model is put to use, it may eventually behave incorrectly in situations that developers did not consider during development or in cases where the training examples were labeled incorrectly. The authors give a clear example: a model designed to detect people who use the road spots a picture of a person printed on a taxi's advertising banner and mistakes it for a real pedestrian. The model's detection is technically correct because the image does show a person, but that person is not real. Other examples include adversarial samples, where a change to the input that people can barely notice causes the model to give a different answer, and cases involving epistemic uncertainty. This type of uncertainty occurs when a model has never encountered anything similar before.
To organize these different situations, the researchers listed the features and properties of corner cases and divided them into three groups. Data-level properties include how new or unusual a sample is, how unclear it is, whether it comes from the real world or was created artificially, its quality and complexity, and how rarely it occurs. Model-level properties include whether the model's decisions can be traced and explained, how difficult they are, whether they can be predicted, how uncertain the model is, how reliable it is, and whether its behavior is consistent. Application-level properties include relevance, coverage, and misspecification, which means that the model's design or requirements do not properly match the task. These properties only make sense when a corner case is considered in relation to a particular model and task. Each property can be viewed from two perspectives: that of a person and that of a machine. These views can differ greatly. For example, a person may easily understand an adversarial sample even though the model cannot interpret it correctly.
The mathematical definition is based on expected loss weighted by relevance. The authors introduce a weighting function designed for each task. It gives each sample a relevance score based on its features, target, and context, such as the distance to an object or the object's class. A corner case is then defined as a relevant sample for which the expected relevance-weighted loss of the best pointwise Bayes predictor is greater than a threshold set in advance. In a simple example involving two classes and inspired by traffic signs, the minimum expected loss is high near the boundary where the model chooses between classes. Looking only at this loss would mark many ordinary samples as corner cases. At the same time, it could overlook important corner cases farther away from that boundary. Combining the relevance-weighting function with the minimum expected loss solves this problem by identifying the samples that truly matter for the task.
This definition works differently at different stages of machine learning. During training, it highlights cases where epistemic uncertainty, which can be reduced by learning from more data, is high and the sample is still relevant to the task. This connects the approach directly to active learning, where a model helps select the most useful examples for further training. During testing, the expected relevance-weighted loss identifies samples that the model treats as corner cases. After deployment, the measure can be calculated without knowing the correct answers, helping detect changes in the data and collect new training examples for situations where the model performs least well. The authors emphasize that the model's confidence must be properly calibrated. In an uncalibrated model, its confidence score does not match the actual chance that its prediction is correct. Methods such as Platt Scaling and Temperature Scaling can improve the accuracy of identifying corner cases through the loss measure, as long as the data used for validation follows the same distribution as the data encountered after deployment.
Traditional performance measures, including accuracy, F1-score, mean squared error, intersection over union, and mean average precision, treat all samples equally. Because of this, they cannot measure how important a corner case is for a particular task. The authors argue that meaningful numerical assessment requires relevance measures designed for the task at hand. They point to a relevance measure for pedestrian detection that uses the distance to an object as an indicator of its importance. An object's class and location, along with measures of complexity, can also help determine relevance. A carefully chosen weighting function combines these indicators with the expected relevance-weighted loss to give each corner case a score that can be adjusted to the needs of a specific application.
The study also expands an existing system for classifying corner cases. The earlier system included a sensor layer, covering hardware and physical factors; a content layer, covering the domain, objects, and scenes; and a temporal layer, covering scenarios. It also included a method layer for errors caused by the machine learning method itself. The new study adds three levels to the method layer, based on the stages of model development. The input level covers corner cases caused by the data, including biased data, uneven numbers of examples across classes, incorrect labels, aleatoric uncertainty, and adversarial noise. Aleatoric uncertainty is uncertainty caused by the data itself and cannot simply be removed by collecting more examples of the same kind. The model level covers errors caused by the model's uncertainty, design, and reliability. The authors explain that the chosen architecture, or the way a model is built, creates an inductive bias: built-in assumptions that can themselves lead to corner cases. The deployment level covers differences between what the model was designed to do and the environment in which it actually operates. For example, incorrect calibration based on a dataset with a different distribution, or a change in the relationship between inputs and correct answers between training and real-world use, can lead to extremely dangerous predictions.
The authors conclude that the many different properties of corner cases should not be underestimated. Some of these properties only become clear when a particular model or application is taken into account. They explain that their definition makes corner cases easier to identify and measure. It offers a way to understand how a model behaves, how it can improve by learning from corner case samples, and whether efforts to reduce risk are working. Future research will implement and test the approach in applications such as object detection and semantic instance segmentation, which involves identifying objects and assigning labels to their pixels. The researchers also acknowledge that it remains an open question how closely the numerical assessment matches the definition itself. They argue that their framework helps clarify which factors matter most when testing and validating machine learning models used in safety-critical systems.
Subject of Research: A mathematical definition and taxonomy of corner cases in machine learning processes
Article Title: Corner cases in machine learning processes
Article References: Heidecker, F., Bieshaar, M., & Sick, B. (2024). Corner cases in machine learning processes. AI Perspectives & Advances, 6(1), Article 1. https://doi.org/10.1186/s42467-023-00015-y
Image Credits: AI Generated
DOI: 10.1186/s42467-023-00015-y
Keywords: machine learning, corner cases, autonomous driving, uncertainty quantification, model calibration, adversarial samples, out-of-distribution data, taxonomy, active learning, data quality, Corner, cases