Service Degradation - Manual Tasks

Summary

Manual task assignment for all EU customers stopped working between 10h50 UTC and 11:50 UTC. This led to an increase in manual processing Turnaround Time (TaT) affecting approximately 20% of our document verification volumes with all customers recovering to TaT SLA by 18h00 UTC.

During this period:

• All checks that required manual review showed an increase in TaT.
• Fully automated reports were not affected and continued to run as normal.

The issue was caused by a configuration error in our internal task management system, which prevented it from correctly assigning tasks to our analysts.

We fixed the configuration and restored normal processing by 28 Jan 2026 11h50 AM UTC, and cleared all manual task backlogs by 18h00 UTC.
We have updated our validation and deployment checks to prevent similar issues in the future.

Root Causes

Manual processing queue assignment was affected by an invalid manual configuration input. This single queue configuration parameter resulted in an error that affected assignments in all queues.

Timeline

• all times in UTC:

10:50: Configuration manually updated and errors started, no more tasks assigned.

11:02: On-call is notified of a spike in manual system assignment errors through our monitoring

11:07: The error responsible for the spike is identified (invalid UUID)

11:10: Incident declared

11:45: Origin of invalid UUID is found

11:50: Bad configuration parameter is deleted

11:56: Configuration reintroduced correctly

18:00: Recovered from manual task backlog

Remedies

• Adding appropriate input configuration value validation
• Improve task assignment resilience to these types of errors
• Review configuration guidance and post-release monitoring