Risk
AI systems can sometimes produce incorrect or made-up information, and they may present these outputs with a high level of confidence. This risk may be greater in situations where there is limited or incomplete data, as the system attempts to fill in gaps rather than signal uncertainty. In contrast, when more reliable information is available, AI may provide more accurate and nuanced responses. For people with disabilities, this creates a heightened accessibility risk. For example, someone who relies on AI-generated image descriptions or summaries may not be able to independently verify whether the information is correct. In these cases, the assistive tool itself can become a source of misinformation, which may lead to confusion, unsafe decisions, or reduced trust in essential supports.
Mitigation
Confidence and uncertainty signals should be required on disability-related outputs, and high-stakes queries (i.e., clinical, legal, or accommodations-related) should be routed to verified resources. A review by an adequately trained human-in-the-loop with domain-specific expertise should be mandatory for any AI-generated information used in safety or medical contexts, or the AI should be explicit in indicating that it is unable to provide accurate information and refer the individual to a human-based source that they may be able to contact for support (i.e., hospital, 911, etc.).
Illustrative Examples
Education
Misleading AI explanations presented with confidence
Students who rely on AI-generated descriptions of images, diagrams, or charts may receive inaccurate or incomplete descriptions. If the student cannot independently verify the content, this may affect understanding of key material, especially in subjects like science or mathematics.
Employment
Confident but incorrect HR guidance
Human resources chatbots may provide incorrect information about policies, workplace rights or accommodation processes while appearing confident. Employees who rely on these tools may not know the information is wrong. This can lead to missed supports or misunderstandings about available options.
Healthcare
Incorrect medical information and advice
Symptom-checker chatbots may sometimes give incorrect answers while appearing very confident. They may miss or misinterpret disability-related symptoms, or suggest conditions that sound believable but are not accurate. This can lead to confusion or delay in seeking the right care.
Services
Inaccurate service guidance
Chatbots used for public services may provide incorrect information about eligibility or processes while appearing confident. People who rely on these tools may not know the information is wrong. This can lead to missed benefits or incorrect applications.