Tagging and flagging
Content moderation is a form of established tagging and flagging methods that advanced analytics experts have developed for other applications. Systems using this approach assess information to identify messages that meet certain criteria, then refer them to specialists who can verify actionable characteristics. Marketing specialists already use these techniques to identify content that mentions their products and services. Social scientists use them to track emerging behaviors and trends. Politicians use them to monitor voter concerns.
As the field evolves, five types of content moderation have emerged.
Types of content moderation
- Pre-moderation – the practice of clearing for approval UGC material before posting it. While this reduces institutional risk, the delay also reduces user satisfaction.
- Post-moderation – the practice of posting UGC content immediately while putting copies of it in a queue for moderators to approve. This improves user satisfaction but increases the risk to institutions that offensive postings or misinformation will be seen.
- Community moderation – This process depends on user communities to identify material that violates an institution's standards and is often used in conjunction with pre- and post-moderation.
- Distributed moderation – this is a self-policing process that relies on the online community to determine what is acceptable content. Because this leaves institutions open to legal and reputational risks, some prefer an in-company distributed moderation system.
- Automated moderation – the process of using digital tools such as AI that apply defined rules to reject or approve user submissions.
All the types listed here, however, can be combined. In fact, a content moderation system can actually be less complex than many of these other tagging and flagging applications. Put simply, it consists of processes that set and enforce policies, along with a governance framework to manage potential bias and assure the safety of moderators. Operational excellence, including robust quality assurance, is also essential.
Here are the components of a successful content-moderation program:
Delivering trustworthy and safe information is the overarching goal, but much depends on platform context and the market in which it operates. Each channel needs tailored policies that establish what's acceptable, so harmful content can be efficiently controlled, leaving legitimate information to flow freely. These policies must be clear and practical to enforce.
They must also be responsive to stakeholder expectations – and that calls for a feedback loop encompassing content creators, moderators, and others. The reason: constant feedback ensures that everyone involved understands the policies and has a voice in influencing them. Qualified teams need to evaluate this feedback, along with other information flowing from the system, to keep the policies and enforcement mechanisms aligned with changing user behavior.
AI plays an important role in detecting policy violations because it can quickly process vast amounts of information. AI can also help verify content producers, since trust and safety depends in part on who originated the information. But this is a highly sensitive and nuanced environment, so well-trained human recruits must still carry much of the burden. Their skill sets will vary, however, depending on the context. For example, the need for language proficiency and knowledge of cultural expression depends on the type of information involved.
With the scale of operations potentially running into thousands of employees, effective risk management requires solid governance. While individual initiative is expected, the system must function within a formal hierarchy of authorities, supported by close monitoring and reporting. This kind of governance ensures that the system meets its goals, which include guarding against unintentional bias that can lead to reputation disaster.
The governance framework should also protect the welfare of moderators. The daily job of reviewing and acting on unpleasant or offensive material can cause emotional harm, violating the employer's duty-of-care obligations and driving attrition. So controls must be put in place similar to those used for other health-and-safety hazards. This includes monitoring employee wellness and provision of counseling, as well as work rotation, systematic breaks, as all tailored to specific operational, policy, and cultural environments.
As with other mission-critical business systems, designing the right operating model for content moderation is essential. If the content is highly sensitive, companies might need 24x7 coverage so that enforcers can deal with potentially malicious material in near real time. In other settings, 24 hours may be an acceptable timeline for removing objectionable content. Either way, automation can optimize the process.
Operational excellence also demands a constantly evolving feedback loop. Is the enforcement team consistently making the right decisions? Should independent quality analysts review decisions? Are other changes needed, such as having more than one enforcer involved in every decision? Again, much depends on the nature of the information and cultural nuances.