Risk
AI systems used in schools, workplaces, healthcare, and public services often collect information about disability under the goal of providing support. However, this same data may also be used in ways that lead to targeting, exclusion, or manipulation. This creates a difficult trade-off for people with disabilities. On one hand, sharing disability-related information can increase the risk of data misuse, privacy breaches, or discrimination, including being singled out or re-identified due to the uniqueness of their data. On the other hand, choosing not to share this information may result in losing access to needed services, accommodations, or support, since many systems rely on disclosure. It can also lead to invisibility, where systems assume a more uniform population because people avoid sharing this information. As a result, the AI may fail to recognize a range of needs and become less effective at supporting people with disabilities.
Mitigation
Purpose-limitation should be enforced in law; disability data collected for support cannot be repurposed. Opt-out from data collection should not result in loss of service access, and independent disability-led oversight bodies should be established. Organizations should put in place processes for recourse and recovery in cases of data abuse.
Illustrative Examples
Education
Student data that risks identifiability
Systems that track student engagement may collect detailed data about behavior and accommodations. Because some disability-related patterns are unique, this data may make students easier to identify, even when it is meant to be anonymous. This can create privacy risks.
Employment
Workplace data that reveals more than intended
Workplace systems that track employee activity may collect detailed data about accommodations or working patterns. Because some of these patterns are unique, they may make individuals easier to identify, even when data is meant to be anonymous. This information may then influence management decisions in ways that are not always clear or fair.
Healthcare
Easily re-identifiable health data
Wearable devices and monitoring tools can collect ongoing health data. Even when this data is labeled as anonymous, it may still be possible to re-identify people with rare conditions because their patterns are unique. This may create privacy risks that are not always clear to patients.
Services
Anonymized service data at risk of re-identification
Systems that track service use may collect detailed information about behavior and accommodations. Because some patterns are unique, this data may make it easier to identify individuals, even when intended to be anonymous. This can raise privacy concerns and affect how people are treated.