Scheduled Notification Delays
Incident Report for Airship
Postmortem

Between approximately 5:38 AM and 7:53 AM on 1 January 2021 US Pacific time, the service responsible for introducing scheduled messages into the sending pipeline on our US cloud site experienced an issue which resulted in up to 56 minute delays. These delays were only experienced on sends previously scheduled to go out on our US cloud site during the incident window. No other sends were impacted by this issue, and we apologize for any inconvenience. The issue was caused by a large number of messages with large personalized payloads being scheduled around the same time. Our Engineering team still plans to rearchitect the system in question in the coming weeks, which should improve its ability to manage this level of workload. There are also several other avenues they can pursue if these changes do not lead to the desired level of performance. In the meantime, our Engineers have increased system capacity and added new metrics and logs so we can be notified of and resolve similar problems more quickly in the future.

Posted Jan 05, 2021 - 07:40 PST

Resolved
This incident has been resolved.
Posted Jan 01, 2021 - 07:48 PST
Update
Scheduled notifications are no longer experiencing delays and this issue has been resolved.

If you have any questions or concerns please contact Airship Technical Support at support.airship.com
Posted Jan 01, 2021 - 07:47 PST
Investigating
We are currently investigating an issue that is resulting in delays for notifications sent using our scheduled notification feature. Notifications that are not scheduled are unaffected by this delay and are performing normally.

If you have any questions or concerns please contact Airship Technical Support at support.airship.com
Posted Jan 01, 2021 - 06:44 PST
This incident affected: Mobile Messaging, Web Notifications, Email Messaging, SMS Messaging, Message Center Messaging, and Open Channels Messaging.