Review Automoderation Tool

This information was sourced from "Review Automoderation Tool" by AI.GOV.UK, https://ai.gov.uk/knowledge-hub/tools/review-automoderation-tool/?directTo=library. Licensed under Open Government Licence v3.0.

The challenge

The NHS.UK website receives approximately 100,000 reviews annually, each of which must be carefully reviewed to ensure compliance with strict moderation policies concerning personal data, abuse, discrimination, and safeguarding. This process depended on third-party human moderators, leading to delays of several days and significant operational costs. Moreover, in 2021, 90% of reviews rejected during moderation were never resubmitted, resulting in the loss of valuable patient feedback.

The solution

An automated moderation tool was developed to enforce nine distinct moderation rules. It combines deterministic checks, such as detecting URLs, email addresses, and profanity, with custom machine learning models trained on NHS.UK content. Reviews flagged for safeguarding concerns or disputed by users are escalated to human moderators. This ensures expert oversight where necessary. The system provides instant feedback to users, allowing them to edit and resubmit reviews promptly.

The results

• Moderation time reduced from several days to just seconds, enabling near-instant feedback to users.
• Increased publication rates, with more reviews successfully published due to immediate detection of policy breaches, unlike the previous manual process where many rejected reviews were never resubmitted.
• Significant cost savings achieved by eliminating reliance on expensive third-party moderation services, while enabling efficient scaling as review volumes grow.
• ML models match or exceed human performance, delivering consistent and accurate policy enforcement across all reviews.
• First AI product to receive clinical approval on NHS.UK.