Increased latency on document check processing

Incident Report for Onfido

Postmortem

Summary

On Nov 2nd 2023, from 17:17 UTC until 18:31, Onfido suffered from an increase in TaT for document reports that needed manual extraction. This issue was focused on manual extraction from USA driving licences.

Root Causes

A release on our manual processing tool to improve how we extract information from the before-mentioned document led to an increase in errors due to a package update that broke our manual processing tool.

Timeline

2023-11-02 17:42 UTC: The first alert is triggered due to the decrease in manually processed document reports;

2023-11-02 18:11 UTC: We identified the issue of not manually processing the USA DLs is related to the latest release of the tool;

2023-11-02 18:19 UTC: We revert to the previous version to re-establish and mitigate the issue;

2023-11-02 18:23 UTC: The issue is confirmed to have been mitigated;

2023-11-02 18:51 UTC: The incident is closed;

Remedies

The following action items will be taken to avoid such issues from happening again:

Review existing monitor and configure it to trigger on-call in case a high amount of manual tasks are not being completed;
Move end-to-end tests from after to before package publishing to block the rollout of such releases.

Posted Nov 28, 2023 - 17:00 UTC

Resolved

This issue is now resolved:

Increased latency on document check processing

We take a lot of pride in running a robust, reliable service and we're working hard to make sure this does not happen again. A detailed postmortem will follow once we've concluded our investigation.

Posted Nov 02, 2023 - 18:50 UTC

Monitoring

We have implemented a fix for this issue.

We are monitoring closely to make sure issue has been resolved and everything is working as expected. Please bear with us while we get back on our feet and we appreciate your patience during this incident.

Posted Nov 02, 2023 - 18:31 UTC

Investigating

We're currently experiencing issues that are negatively impacting latency on document check processing.

Our next update will be in 15 minutes. Thank you for your patience.

Posted Nov 02, 2023 - 18:21 UTC

This incident affected: USA (us.onfido.com) (Document Verification), Canada (ca.onfido.com) (Document Verification), and Europe (onfido.com) (Document Verification).