Amazon's artificial intelligence, designed to sort job seekers' resumes, ranked men higher than women. In particular, he rejected CVs with the word “feminine” and offered to hire candidates who used typical masculine verbs in their speech, even if these people did not have the necessary skills. As a result, the company had to close the project.
Artificial HR intelligence
Amazon's machine learning experts have found that the artificial intelligence that processes the resume of candidates for positions at the company discriminates against female job seekers, Reuters reported. This feature was not intentionally incorporated into the system - it was the result of machine learning.
An artificial intelligence-based resume sorting engine has been under development at Amazon since 2014 by a 12-person team in Edinburgh. Already in 2015, the company noticed that the new system did not adhere to the principle of gender neutrality when evaluating candidates for software development positions and other technical vacancies.
Soon they realized the reason: the fact is that the system was trained on the resumes entered by the company over the past 10 years. Most of these resumes were held by men, which broadly reflects the state of affairs in the IT industry. As a result, Amazon's artificial intelligence learned that male job seekers are preferred over women.
Linguistic rejection
Promotional video:
As a result, artificial intelligence began to discard resumes containing the word "female" in such expressions as "captain of the women's chess club." The mechanism also downgraded the grades of graduates of two women's colleges, but Reuters sources did not specify their names. When evaluating resumes, artificial intelligence gave them between one and five stars - very similar to how buyers rate products on Amazon.
The system paid attention to other words as well. The developers have created 500 computer models for different job responsibilities and geographic locations. Each model was trained to recognize about 50 thousand signal words that were present in the applicant's resume.
It is noteworthy that the mechanism ignored words denoting skills that are often spelled out in the resumes of IT professionals, such as the ability to write various codes, etc. Instead, the system tracked, for example, typical verbs for male speech in the self-description of candidates, like "fulfilled" (executed) or "captured", as they were regularly seen in old resumes that the machine learned from. With this approach, gender discrimination was not the only problem - the mechanism often recommended people who did not have the necessary skills for the position.
The further fate of the project
Amazon has adjusted the system so that it no longer responds to signal words. However, this was not a guarantee that the mechanism would not find another way to identify resumes belonging to women. By early 2017, the company became disillusioned with the project and disbanded the team. Reuters sources say the HR specialists took into account the system's recommendations but never relied solely on them.
One of the sources says that the project is closed. Another claims that a new team was formed in Edinburgh to continue developing the mechanism, but this time with a focus on equity and inclusiveness. Another source reports that a heavily modified version of the engine is now being used to perform routine work, such as removing duplicate resumes from the database.