Risk
AI tools may appear to remove a barrier, but the tool may address the needs of some people better than others. Their effectiveness can vary depending on a person’s needs, characteristics, or context. Because these systems are often designed around common patterns, they may perform less well for people whose situations are different. As a result, some people with disabilities may receive less benefit from tools that are intended to help them, and appear to work well overall. For example, speech recognition may not work reliably for people with different speech patterns, and wearable health tools may miss important signals in disabled bodies. This can create a false sense of progress, where organizations assume the problem has been solved and reduce human support, even though the technology does not meet everyone’s needs.
Mitigation
Performance reporting should include worst-case subgroup accuracy, not only average accuracy. Human alternatives should remain available wherever AI is deployed.
Illustrative Examples
Education
Learning access tools that don’t work for those who need them most
Speech recognition tools may be introduced to support students who prefer speaking over typing. However, they may not work reliably for students with different speech patterns. This can make it harder for those students to complete assignments, even though the tool is meant to improve access.
Employment
Variation in accessibility effectiveness in workplace tools and practices
A company may introduce automated captions in meetings and events and assume that captioning needs have been fully addressed. However, automated captions may contain frequent errors, missed words, or incorrect terminology, especially in fast-paced or technical discussions. As a result, Deaf and hard of hearing employees may still not be able to fully follow the conversation. Despite this, the company may reduce or stop providing human captioning services (such as Communication Access Realtime Translation, or CART), creating a situation where access appears to be improved but is less effective for the people who rely on it most.
Healthcare
Monitoring tools that produce uneven accuracy
Some health monitoring tools may not measure vital signs as accurately for people with different body types, movement patterns, or assistive devices. This can lead to less reliable data for people with disabilities. If clinicians rely on this data, decisions may not reflect the patient’s actual condition.
Services
Digital access replacing more effective human support
A government service may introduce a chatbot to help people apply for benefits or get information, and assume this improves access. However, while the chatbot can be helpful for some people with disabilities, it may not perform as well in more complex cases. As a result, people with disabilities who are more extreme outliers may receive incomplete or unclear answers. If access to human support is reduced because the organization believes they are addressing access needs through the chatbot, this can make it harder to get accurate guidance, even though the service appears to be more accessible.