Delays in check processing

Incident Report for Onfido

Postmortem

Summary

On 12th Feb 2024, from 03:57 GMT to 04:36 GMT a fraction of facial similarity checks and of document checks requiring manual review for some customers in the EU region experienced an increase in turnaround time. Delayed checks were then processed and at 05:15 GMT the system was back to normal.

Root Causes

An erroneous configuration was released on 9th Feb 2024 for a service responsible for handling checks requiring manual review leading to most replicas of the service to eventually fail.

Timeline

03:57 GMT: Decrease in throughput of manual reviews in EU.

04:05 GMT: The On-call team was alerted to an increase in error rate on a specific service.

04:36 GMT: Fix is applied and system was back to normal operation.

05:15 GMT: Completion of all backlog of checks generated during incident window.

Remedies

Reverted the erroneous configuration.

Review internal documentation for preventing that from happening.

Reviewed monitoring for alerting Engineering more promptly.

Posted Feb 15, 2024 - 11:17 UTC

Resolved

This issue has been resolved and we've applied a fix. We expect to complete the remaining delayed checks in the next 45 minutes or so. We apologize for any inconvenience this has caused.

Posted Feb 12, 2024 - 05:02 UTC

Monitoring

We are monitoring.

Posted Feb 12, 2024 - 04:43 UTC

Investigating

We are currently investigating this issue.

Posted Feb 12, 2024 - 04:39 UTC

This incident affected: USA (us.onfido.com) (Document Verification, Facial Similarity), Canada (ca.onfido.com) (Document Verification, Facial Similarity), and Europe (onfido.com) (Document Verification, Facial Similarity).