Risk
AI replicates and amplifies historical patterns of exclusion. Persons with disabilities have long been excluded from employment, education, and services; AI screening tools learn these exclusion patterns from their training datasets and continue to perpetuate them (e.g, by filtering out applications that include direct or indirect indicators of disability). Machine learning can easily learn and pick up on these patterns, even when disabilities are not explicitly disclosed.
Mitigation
Design AI systems to prioritize exploration, not just past patterns. Instead of relying only on historical data to make decisions (for example, selecting applicants who look similar to past “successful” candidates), systems can be tuned to actively consider a wider range of profiles and experiences. This means giving fair attention to applicants with non-standard paths, such as gaps in education or work that may reflect disability-related factors. Human review of negative decisions and the creation of a clear appeals pathway should be mandatory, and indirect disability proxies should be prohibited as model features.
Illustrative Examples
Education
Discrimination against disability in admissions
Admissions systems that rely on patterns such as continuous enrollment may disadvantage students whose records reflect disability-related disruptions. These systems may categorize applicants who have taken medical leave as less academically strong even when the gap in studies is not related to student performance.
Employment
Hiring filters that screen out non-traditional work histories
Résumé-screening systems may use patterns in work history to filter candidates, which can disadvantage applicants with gaps due to medical leave or disability-related needs. Even when disability-based discrimination is not allowed, these systems may still pick up indirect signals and exclude candidates. As a result, qualified applicants may be overlooked.
Healthcare
Resource allocation based on biased historical care
Care-prioritization systems may use past data to decide how to allocate resources. Because people with disabilities have sometimes received less care in the past, these systems may underestimate their needs. This may result in fewer services or lower priority, even when support is needed.
Services
Past exclusion patterns shaping future eligibility
Eligibility systems may use patterns from past applications to make decisions. If people with disabilities were previously denied at higher rates, these systems may continue to reflect those patterns. This can result in applicants being excluded based on indirect signals rather than their actual eligibility.