System latency
Incident Report for Sertifi
Postmortem

A hardware failure caused a storage controller failover on one of our server chassis early Thursday (7/15/21) morning. Servers recovered and no latency was occurring due to low load on the agreement platform. Once load increased around 8:15am, more underlying storage issues were occured. We moved production servers to another  cluster to mitigate any issues while working with our hardware provider to resolve the latency. Latency was resolved with a firmware update to a certain set of hard drive model that is present in our cluster as well as the chassis firmware.

Posted Jul 22, 2021 - 12:02 CDT

Resolved
This incident has been resolved.
Posted Jul 16, 2021 - 08:59 CDT
Update
System performance has returned to normal, we will continue to monitor
Posted Jul 15, 2021 - 12:05 CDT
Update
Sertifi systems are still slower than normal while the backlog of events are cleared. We will update when all functionality returns to normal.
Posted Jul 15, 2021 - 11:10 CDT
Monitoring
We have identified the issue and implemented a resolution; our team is continuing to monitor the situation.
Posted Jul 15, 2021 - 10:31 CDT
Update
Latency issues are causing problems system wide, but are especially impacting API integrations and Salesforce integration customers.
Posted Jul 15, 2021 - 09:55 CDT
Investigating
We are currently investigating reports of system slowness.
Posted Jul 15, 2021 - 09:24 CDT
This incident affected: Web Portal, Infrastructure, API, Salesforce Integration, eConfirmations, Email, Payment Systems, and eAuthorizations.