Push delivery delays and issues accessing to the Go Dashboard
Incident Report for Airship
Postmortem

Our platform experienced several issues today due to problems experienced by our third party cloud provider.

First, between about 2:46-3:00AM US Pacific time, our push API was returning a high number of errors, making it difficult to send or schedule push messages. Our web dashboard also returned a variety of errors during this period.

By about 3AM, push API and critical website functionality had improved. Between about 3:00 and 3:30AM US Pacific time, there were significant delays in delivering messages and several other APIs continued to return a large number of errors.

Finally, between about 3:30 and 10:30AM Pacific, all other issues had resolved except for a few device registration/lookup APIs and systems still exhibiting significant problems or delays. This also impacted the ability of customers to modify test lists.

Around 10:30AM, we were able to implement a final workaround for remaining issues still being experienced by our third party cloud provider. We sincerely apologize for the inconvenience, and we are both awaiting an RCA from our provider and working internally on improving our methods for troubleshooting, communication, and developing workarounds for similar issues in the future.

Posted Nov 13, 2019 - 14:02 PST

Resolved
We have resolved all service issues related to the outage. The channel registration service is now fully functional but is experiencing a slight delay due to a large amount of traffic being processed at this time. Messaging on all channels is working normally.

If you have any questions or concerns, please contact Airship Technical Support at https://support.airship.com/
Posted Nov 11, 2019 - 11:49 PST
Update
We are continuing to work on restoring our channel registration services. Any unprocessed changes will be automatically processed once the channel registration service is restored to full service.

The previous issues relating to sending push through the API and dashboard are resolved and working normally.

If you have any questions or concerns, please contact Airship Technical Support at https://support.airship.com/
Posted Nov 11, 2019 - 08:34 PST
Update
We are continuing to work on a fix for this issue.
Posted Nov 11, 2019 - 06:51 PST
Update
Our third party provider has recovered. Push notification services through the API and Dashboard have been restored, and our engineers are actively working to restore full functionality to the channel registration service. At this time, new and existing channel registrations are currently delayed. These will be processed when the channel registration service recovers.

We are continuing to work on resolving the channel registration issue impacting our system.

If you have any questions or concerns, please contact Airship Technical Support at https://support.airship.com/
Posted Nov 11, 2019 - 06:50 PST
Identified
We have identified the issue in which our systems are impacted by a partial outage with a third party service.

Our engineering team is monitoring the issue and working to restore our systems.
Posted Nov 11, 2019 - 04:56 PST
Investigating
We are currently Investigating issues with delays when sending messages either through the Message Composer or the API with some 500 errors returned for API calls. Access to the Go Dashboard has been affected as well.

Our engineering teams are working to identify and resolve this issue. We will update our status page as soon as we have more information.

If you have any questions or concerns, please contact Airship Technical Support at https://support.airship.com/
Posted Nov 11, 2019 - 03:49 PST
This incident affected: Web Dashboard, APIs, Mobile Messaging, and Web Notifications.