Risk
Automated risk detection and flagging systems (e.g., exam proctoring, airport security, fraud detection, workplace monitoring) are typically based on the behaviours and bodies of average, able-bodied individuals. As a result, disabled bodies, behaviours, and accommodation devices (assistive technologies) are disproportionately flagged as suspicious, or fraudulent.
Mitigation
Pre-deployment testing should include checks for false positives in disability-related scenarios. Appeals processes should always be available, and not depend on disability disclosure. The use of risk detection AI systems in critical access settings (i.e., exams, border crossings, and social protection) should only be used alongside human supervision to help reduce incorrect flagging. However, human reviewers are often required to make decisions under tight time limits and heavy workloads, which may not allow for careful judgment and can limit the effectiveness of this safeguard.
Illustrative Examples
Education
Disability-related behaviour flagged as cheating
Online exam proctoring systems may flag certain behaviors or tools as suspicious, even when they are related to disability. For example, assistive devices, different eye movements, or physical differences may be misinterpreted as signs of cheating. This can lead to unfair scrutiny or penalties.
Employment
Workplace monitoring misinterprets disability-related activity
Workplace monitoring systems may flag certain behaviors as unusual or suspicious, even when they are related to disability. For example, taking breaks for health reasons or using assistive technology may be marked as low productivity or potential misuse. This can lead to unfair scrutiny or incorrect performance assessments.
Healthcare
Disability-linked care patterns flagged as suspicious
Insurance fraud detection systems may be more likely to flag certain patterns linked to disabilities as suspicious. Because people with disabilities often have ongoing care needs, multiple conditions, or complex treatment histories, their claims may be treated as unusual or higher risk. This can lead to claims being denied or flagged for review more often, creating barriers to care, delays in treatment, added administrative burden, and increased stress for patients who may already be managing significant health needs; in serious cases, these delays may disrupt essential care and contribute to worsening health or life-threatening outcomes.
Services
Security systems misreading disability-related behaviour
Systems used in areas such as border security or fraud detection may flag certain behaviors or tools as suspicious. People with disabilities, including those using assistive devices or with different movement patterns, may be more likely to be flagged. This can lead to delays, additional checks, or stress.