Osano dashboard login issues

Incident Report for Osano

Postmortem

Osano relies heavily on Amazon Web Services for all infrastructure. Cognito which is fault tolerant powers the authentication into the Osano dashboard experienced a major outage for approximately 6 hours.

Osano engineering confirmed that the errors were not due to Osano configuration issues but rather were caused by Cognito having an outage. The Cognito outage was the result of failures on a heavily used streaming component on Amazon called Kinesis. Kinesis had a global outage across many data centers that impacted AWS Cognito along with numerous other AWS services.

Outages of Kinesis on this scale are extremely rare and Cognito has had nearly 2 years without a single incident, so while this outage was inconvenient to customers, it is unlikely to recur and AWS has reassured us that mitigations are now in place to prevent this issue in the future.

Posted Nov 25, 2020 - 18:00 CST

Resolved

Customer authentication issues have been resolved. The root cause of this outage was due to Osano's reliance on AWS Cognito identity stores which began experiencing increased API failure rates due to an issue with Kinesis Data Streams. AWS have implemented a mitigation to this issue.

Posted Nov 25, 2020 - 17:57 CST

Monitoring

Osano is currently experiencing an outage affecting the ability to log into the Osano dashboard. This outage does not impact visitor facing services such as consent management and subject rights management. The root cause is Osano's reliance on Amazon Web Service's Cognito authentication solution for processing customer credentials in the web app. The engineering team is monitoring the situation and will update this status as soon as it is resolved.

Posted Nov 25, 2020 - 11:19 CST

This incident affected: Platform (Customer Authentication).