On Sunday 1st July 2018 at 23:15, the Onfido UK Identity Record, UK Credit, and UK Address Picker services went down. Service was resumed for UK Identity Record on Monday 2nd July at 11:44, and UK Credit and UK Address Picker services were resumed on Tuesday 3rd July at 09:44.
In keeping with our commitment to transparency, the following is a report of the issue we encountered, the factors that contributed to that issue, and ultimately, what we’ve done and plan to do to ensure we don't find ourselves in this situation again.
Every 15 days, we trigger a password reset process to continue accessing the services of one of our third party data providers. There was a bug in the code which attempted to repeat this process, but one of our backend services failed to update to the second new password. As a result, we were requesting our data provider with two different passwords (one correct, one incorrect), and our account was suspended. This caused a full outage for the respective services.
Onfido leverages multiple data providers for some of the affected services, and normally we would be able to automatically revert to a secondary, or back-up, provider. In this instance, our fallback logic was also behaving in an unexpected manner and we were unsuccessful in reverting to alternatives.
We corrected the fallback logic for our Identity Record check at 11:44 on Monday 2nd July, and were able to restore this service. We will continue to explore options to build redundancy into our systems. We fixed the original issue with the password reset process at 09:44 on Tuesday 3rd July by resetting and reverting to our original password, and have since removed the duplicate logic.
The password reset process has also been rescheduled to run every 14 days during business hours, and we have successfully executed a password reset since the event.
We will continue to work closely with all of our data providers to ensure rapid identification and resolution of any issues.