AI Inequalities: Ethnicity
The use of AI in workplaces has been found to discriminate against Black, Asian and Minority Ethnic workers based on both direct and indirect forms of discrimination. Although the design of particular AI models may not explicitly use demographic data about ethnicity, other data can serve as precise proxies for such characteristics in ways that result in direct forms of discrimination. Data such as names, languages spoken, and postcodes, have all been found to act as proxies for ethnicity, and research has shown how the use of such data disadvantages workers from ethnic minorities.
Such discrimination can happen early on in the hiring process, for example, with the use of HR chatbots that use a specific database of employee language phrases to guide conversation and that struggle with written and spoken language that is not common amongst the dominant (White) group. In fact, speech recognition tools from all the major technology providers have been shown to have significantly higher error rates with people who are Black. AI-driven analysis of video interviews has similarly been shown to favour mannerisms associated with White males, who tend to make up more of the training data as the dominant group of past employees, while penalising women and ethnic minorities who do not exhibit the same verbal and nonverbal cues.
In general, AI tools have been found to reflect racialised norms including in the use of such tools for assessments or profiling of workers based on voice, facial expressions, or language, with ethnic minorities more likely to be disadvantaged. This has been shown to have a very significant impact because ethnic minorities are simultaneously overrepresented in sectors where data-driven technologies are most intensely used, such as in gig economy work. Ongoing campaigns against the use of facial recognition technology to manage workers doing platform work, most notably at Uber, have highlighted cases of workers from ethnic minorities who have been discriminated against and lost their jobs because the technology has not been able to recognise them. At the same time, some workers from ethnic minority backgrounds, and particularly migrant workers, have been found to prefer employment with a heavy reliance on algorithmic management, such as platform labour, as they feel they are less exposed to direct interpersonal discrimination. This preference, research suggests, might be at the cost of other forms of more structural and indirect discrimination happening through AI tools.
Although research on discrimination against Black, Asian and Minority Ethnic workers relating to AI use outside of the gig economy is still limited, there are concerns about the use of new types of technology like ‘Emotional AI’ that is being used in the United States, for example, to inform decision-making including about promotion to leadership positions, fitness-to-task, and the construction of teams. This kind of technology tends to struggle with the cultural and context-dependent nature of technologies, often misinterpreting emotions such as anger and contempt, particularly amongst women and ethnic minorities. The growing reliance on Generative AI has also been found to further racial stereotyping and discrimination when used, for example, for ranking potential candidates. For example, in another study in the US, resumés with names distinct to Black Americans were the least likely to be ranked as the top candidate for a financial analyst role, whereas resumés with names distinct to Asian women were ranked as the top candidate for the financial analyst role more than twice as often.
Some have called for greater diversity within the fields of computing and engineering as a response to this kind of discrimination and to try and ensure that a wider range of identities and experiences are taken into account when systems are being designed. Amongst the largest technology companies, such as Google, Meta and Microsoft, there continues to be what some refer to as a diversity crisis in relation to race and gender. But others argue that addressing the lack of diversity in the AI sector is only a small part of the solution, and that as these systems are being adopted across sectors at rapid speed, much stricter auditing of AI tools used in employment is necessary. Yet what such auditing would entail and how it would account for the kind of structural and indirect discrimination that researchers have found to be prominent is not clear. To help the process, the European Network Against Racism has developed a toolkit for mitigating and preventing racial bias in hiring, aimed at HR managers, and Diversity and Inclusion managers, that provides some practical steps for complying with guidelines for the ethical use of AI. No formal evidence has been provided about what impact such a toolkit has had.
TUC Cymru is campaigning for all workers to be protected against the risks of AI. If you’re a Black, Asian or Minority Ethnic worker and concerned about these issues, raise them at your trade union branch. TUC Cymru has successfully negotiated guidance on the use of AI in the public sector. Use it and adapt it for your workplace.
The TUC is campaigning for additional legal protections against the threats of AI and has produced a range of materials to assist reps and officers.
The AI Inequalities at Work report is published on the Data Justice Lab’s website.