Issue affecting credit reports and UK address lookup
Incident Report for Onfido
Postmortem

Summary

On Sunday 1st July 2018 at 23:15, the Onfido UK Identity Record, UK Credit, and UK Address Picker services went down. Service was resumed for UK Identity Record on Monday 2nd July at 11:44, and UK Credit and UK Address Picker services were resumed on Tuesday 3rd July at 09:44.

In keeping with our commitment to transparency, the following is a report of the issue we encountered, the factors that contributed to that issue, and ultimately, what we’ve done and plan to do to ensure we don't find ourselves in this situation again.

Timeline

  • On Sunday 1st July 2018 at 23:15, the Onfido UK Identity Record, UK Credit, and UK Address Picker services went down. This meant that any applicants with a UK address using the Identity Record check, the Credit check, or any other service calling our UK address picker endpoint failed to process.
  • At 11:44 on Monday 2nd July, service was resumed for the UK Identity Record, and we subsequently re-ran all failed applicants on this service.
  • At 09:44 on Tuesday 3rd July, service was resumed for the UK Credit and UK Address Picker, and we subsequently re-ran all failed applicants on UK Credit.

Contributing Factors

Every 15 days, we trigger a password reset process to continue accessing the services of one of our third party data providers. There was a bug in the code which attempted to repeat this process, but one of our backend services failed to update to the second new password. As a result, we were requesting our data provider with two different passwords (one correct, one incorrect), and our account was suspended. This caused a full outage for the respective services.

Onfido leverages multiple data providers for some of the affected services, and normally we would be able to automatically revert to a secondary, or back-up, provider. In this instance, our fallback logic was also behaving in an unexpected manner and we were unsuccessful in reverting to alternatives.

Remediations

We corrected the fallback logic for our Identity Record check at 11:44 on Monday 2nd July, and were able to restore this service. We will continue to explore options to build redundancy into our systems. We fixed the original issue with the password reset process at 09:44 on Tuesday 3rd July by resetting and reverting to our original password, and have since removed the duplicate logic.

The password reset process has also been rescheduled to run every 14 days during business hours, and we have successfully executed a password reset since the event.

We will continue to work closely with all of our data providers to ensure rapid identification and resolution of any issues.

Posted 4 months ago. Jul 23, 2018 - 11:44 BST

Resolved
This incident is now resolved; performance has been stable during our monitoring period.
Posted 4 months ago. Jul 03, 2018 - 12:07 BST
Monitoring
Service has been restored for UK address lookup and credit reports. We're continuing to monitor the fix we've applied and working with our provider to understand the root cause of this issue.

We appreciate that the time taken to resolve this issue was unusually lengthy. Our account management and support teams will follow up with affected customers; we'll also publish a post-mortem once we've completed our investigation. We have also re-run all affected ID and credit record checks that we received during this time period.
Posted 4 months ago. Jul 03, 2018 - 11:16 BST
Update
We're continuing to work with our data partner and anticipate being able to restore service for this functionality within the next 90 minutes.
Posted 4 months ago. Jul 03, 2018 - 10:07 BST
Update
We have discovered that credit reports are affected by the same issue. The UK address lookup API continues to be affected. We are continuing to work with our data provider to resolve this issue. Next update expected at 17 BST.
Posted 4 months ago. Jul 02, 2018 - 15:14 BST
Update
We are continuing to work on a fix for this issue.
Posted 4 months ago. Jul 02, 2018 - 15:13 BST
Update
We are continuing to work with our data provider to resolve this issue.
Posted 4 months ago. Jul 02, 2018 - 12:37 BST
Identified
Service has been restored for UK ID record checks.

We are still experiencing issues with UK address lookups and are working with our data provider to identify and resolve the issue. Next update by noon BST.
Posted 4 months ago. Jul 02, 2018 - 11:19 BST
Investigating
We're currently experiencing an issue affecting the processing of all United Kingdom ID record checks and UK address lookups. The team is investigating. Next update at 11:30 BST.
Posted 5 months ago. Jul 02, 2018 - 10:30 BST
This incident affected: Dashboard and API.