Risk
AI systems rely on statistical reasoning, making predictions or decisions based on what is most common or likely to happen. AI relies on patterns, trends and probabilities, so it is optimized for the average or typical. This can lead to unfair, inaccurate, or wrong decisions for anyone who is far from a statistical average, including persons with disabilities who, by virtue of disability, exist at the statistical edges and vary substantially from the average, and from one another. In addition, risks and harms experienced by disabled persons are often dismissed as statistically insignificant or merely anecdotal because of their uniqueness (see R14).
Mitigation
Disability-disaggregated impact assessments should be required before deployment, and AI tools should be tested against statistical edges as opposed to averages alone. Disability-disaggregated impact assessments alone are insufficient because data-based mitigation strategies replicate the issues of statistical discrimination (see R14). AI should not be relied upon when persons or scenarios are out of distribution in the training data. Reporting worst-case accuracy for subgroups alongside average accuracy helps to ensure that exclusion at the edges is surfaced, as opposed to hidden.
Illustrative Examples
Education
Learning systems built for the “typical” student
Adaptive learning systems are often designed around typical patterns of student behavior. As a result, they may be less effective for students with disabilities whose ways of learning or engaging fall outside those patterns. This may lead to lower-quality feedback, inappropriate difficulty levels, or missed learning needs.
Employment
Performance measured against narrow productivity norms
A workplace performance evaluation system may compare employees against common productivity patterns, such as consistent work hours, typing speed, or communication style. An employee with a disability who works in shorter bursts, uses assistive tools, or communicates in different ways may fall outside these patterns. As a result, the system may rate their performance as lower or less consistent, even when their work quality is strong. These differences may be seen as normal variation in the data rather than a sign of bias, which can prevent managers from recognizing and addressing unfair evaluation practices.
Healthcare
Diagnoses shaped by average symptom patterns
Diagnostic artificial intelligence (AI) systems are often trained on common symptom patterns. As a result, these systems may be more likely to miss or under-detect conditions in people with atypical symptoms or physiology that do not match the data the system was trained on. This can lead to delayed diagnosis or incomplete care when the system does not recognize less common presentations.
Services
Benefit decisions based on average spending patterns
Eligibility systems are often built on common spending and usage patterns. As a result, they may be less accurate for people whose expenses reflect disability-related needs. This can lead to misclassification or incorrect decisions about benefit eligibility.