Web Portals and API are down
Incident Report for Sertifi
Postmortem

On Tuesday, September 20, a Production Deployment of our application resulted in a lockout of a production service account, which blocked user authentication to access our site. A monitor previously generated to alert for these scenarios was too narrowly targeted and did not internally alert of us this incident.

To prevent this from reoccurring, we have taken the following steps:

  • We have restructured the monitor to alert in event of service account lockout. This monitor is categorized as a “Severity 1” and will page the on-call engineer when triggered.
  • We have removed the problem server from our production pool.
  • Next week, we will be redeploying the service which led to the service account lockout onto its own newly provisioned and dedicated server pool.
Posted Sep 23, 2022 - 13:57 CDT

Resolved
Web Portals and API were periodically inaccessible from 9:54am-10:32am.
Posted Sep 20, 2022 - 10:00 CDT