- Case study
Securing a hate-speech-free environment for a tech giant
How automation keeps platform users and workers safe by removing toxic content
One of the world's largest social media companies, with hundreds of millions of users around the globe
Genpact analyzed the platform's user-generated content to identify trends. We evaluated how content moderators rated hate speech, and used artificial intelligence (AI) to filter out unacceptable posts
Challenge
Hate crimes are on the rise around the world. According to The Guardian, hate crimes doubled in England and Wales between 2013-18. The FBI's data for 2019 reveals that hate crimes in the US are at the highest in over a decade. And in 2020, the pandemic brought a surge in crimes targeting Asian-Americans, with the New York Police Department reporting a 1,900% increase.
Social media posts are behind a great deal of this menacing activity. To stop its spread, more countries are introducing stringent social media laws and regulations. Complying is not only obligatory, it's the right thing to do. A media giant knew user-generated content was a potential breeding ground for hate speech because its existing AI system couldn't dependably filter out unwanted posts. As a result, content moderators had to deal with massive amounts of toxic information every day. They also struggled to consistently and clearly evaluate the material they assessed. The situation was urgent, so the company acted.
Solution
We partnered with the company to produce a detailed review of its trust-and-safety framework. We quickly learned that its existing AI wasn't equipped to fully analyze subjective or ambiguous material, and so it missed a great deal of hate speech.
With this insight, we designed and built a new framework. We created an ecosystem to help teams calibrate their ratings more accurately. And we introduced a digital dashboard that tags content based on trends that the new system actively monitors. This simplified process helped the AI learn about potentially objectionable material in real time before the content reached the moderators.
In addition, we improved the moderation process to identify false positives, which removed content in error. To that end, we instituted regular and detailed inspections to check that the process only deletes harmful.
Hate speech is hard on the content moderators who must deal with it every day. The new framework enhances moderators' well-being and resilience by reducing the stress and harmful effects from excessive exposure to challenging content.
Now, Genpact manages 90% of the hate speech that appears on the company's platform in North America.
Impact
Our work to revamp the company's trust-and-safety framework, simplify processes, and improve its use of technology brought benefits to both the platform's users and workforce:
Inappropriate content on the internet is a scourge. With Genpact's help, this social media giant has an effective framework to eliminate it while increasing trust and safety. It's creating a more positive user experience for both visitors and employees.