Summary

A regression was introduced that resulted in some document reports being returned with extracted data missing. This issue was present from 2019-07-16 11:30 to 2019-07-17 13:41 and affected the extraction of vehicle category information from driver’s licenses, a configurable feature that can be requested through the Onfido API.

After a fix for this issue was tested and released, we reprocessed the affected document reports and reached out directly to impacted customers.

Timeline

The following section outlines the timeline of this incident.

2019-07-16

• 11:30: we released one of our backend services, which is used to process and store extracted data. This release introduces a defect which causes vehicle category information to not be persisted during document report processing.

2019-07-17

• 8:53: we were notified of this issue with extraction by an Onfido customer.
• 10:07: the customer issue is escalated to Onfido’s on-call rotation for further investigation
• 12:21: the cause of the issue is identified and work begins on a fix.
• 13:41: the fix is released and is live in production. After this point, no new Document Reports are affected by the regression.
• 13:41: the responding team begins work on producing and testing a script to reprocess affected reports.

2019-07-18

• 17:48: batch re-processing of affected reports begins.
• 19:12: batch re-processing of affected reports finishes and all document reports with missing information are fixed

Root Cause

The issue was caused by a change in the code that selects attributes for persistence upon the submission of extraction information. This change was intended to simplify the logic that selected these attributes.

Unfortunately, this caused us to filter out driving licence specific attributes. As these attributes did not have any specific validation rules applied upon submission, these fields were absent in the database and not persisted upon report completion.

Normally, we would use automated unit tests to make sure we maintain existing behaviour at this level of granularity. However, in this case, test coverage was insufficient; while we did test the attribute selection for persistence, we did not test specifically for the driving licence information, thus this omission was not caught.

Remedies

We take this incident seriously and are conscious of the impact this regression had on some customer workflows. In order to prevent similar incidents from occurring in the future, we have:

• Improved automated tests around this feature, which will prevent similar issues with saving document type-specific extracted data. We have also refactored the underlying code to improve the clarity of the exposed API.

To further improve our ability to detect and avoid impact, we are working on:

• Enhancing the robustness of our data validations so that we are alerted when we are failing to save all the attributes required by the document report configuration.

Lastly, the reprocessing step took longer than anticipated and could have been significantly faster. We have codified the process we used to reprocess document reports to simplify any future response.