Securing a hate-speech-free environment for a tech giant

Challenge

Toxic content impacting users and workers

Hate crimes are on the rise around the world. According to The Guardian, hate crimes doubled in England and Wales between 2013-18. The FBI's data for 2019 reveals that hate crimes in the US are at the highest in over a decade. And in 2020, the pandemic brought a surge in crimes targeting Asian-Americans, with the New York Police Department reporting a 1,900% increase.

Social media posts are behind a great deal of this menacing activity. To stop its spread, more countries are introducing stringent social media laws and regulations. Complying is not only obligatory, it's the right thing to do. A media giant knew user-generated content was a potential breeding ground for hate speech because its existing AI system couldn't dependably filter out unwanted posts. As a result, content moderators had to deal with massive amounts of toxic information every day. They also struggled to consistently and clearly evaluate the material they assessed. The situation was urgent, so the company acted.

Solution

A new trust-and-safety framework with more advanced AI

We partnered with the company to produce a detailed review of its trust-and-safety framework. We quickly learned that its existing AI wasn't equipped to fully analyze subjective or ambiguous material, and so it missed a great deal of hate speech.

With this insight, we designed and built a new framework. We created an ecosystem to help teams calibrate their ratings more accurately. And we introduced a digital dashboard that tags content based on trends that the new system actively monitors. This simplified process helped the AI learn about potentially objectionable material in real time before the content reached the moderators.

In addition, we improved the moderation process to identify false positives, which removed content in error. To that end, we instituted regular and detailed inspections to check that the process only deletes harmful.

Hate speech is hard on the content moderators who must deal with it every day. The new framework enhances moderators' well-being and resilience by reducing the stress and harmful effects from excessive exposure to challenging content.

Now, Genpact manages 90% of the hate speech that appears on the company's platform in North America.

Impact

Consistent ratings, safer users, and healthier workers

Our work to revamp the company's trust-and-safety framework, simplify processes, and improve its use of technology brought benefits to both the platform's users and workforce:

• More consistent, accurate ratings – We helped content moderators achieve 99% consistency levels
• Effective AI – A more robust framework that allows AI to better manage and filter content. With less human intervention, content moderators work in a safer environment with reduced exposure to toxic content, and platform visitors have an improved user experience
• Accurate content tagging – Identifying foul content became easier with a more effective AI and continuous machine learning. We delivered a 6 % increase in correctly identified hate speech
• Employee well-being – A positively charged work environment through protective and corrective approaches to employee safety

Inappropriate content on the internet is a scourge. With Genpact's help, this social media giant has an effective framework to eliminate it while increasing trust and safety. It's creating a more positive user experience for both visitors and employees.