A new trust-and-safety framework with more advanced AI
We partnered with the company to produce a detailed review of its trust-and-safety framework. We quickly learned that its existing AI wasn't equipped to fully analyze subjective or ambiguous material, and so it missed a great deal of hate speech.
With this insight, we designed and built a new framework. We created an ecosystem to help teams calibrate their ratings more accurately. And we introduced a digital dashboard that tags content based on trends that the new system actively monitors. This simplified process helped the AI learn about potentially objectionable material in real time before the content reached the moderators.
In addition, we improved the moderation process to identify false positives, which removed content in error. To that end, we instituted regular and detailed inspections to check that the process only deletes harmful.
Hate speech is hard on the content moderators who must deal with it every day. The new framework enhances moderators' well-being and resilience by reducing the stress and harmful effects from excessive exposure to challenging content.
Now, Genpact manages 90% of the hate speech that appears on the company's platform in North America.