Go Dashboard and API Availability
Incident Report for Airship
Postmortem

Between approximately 1:50-7:43 AM US Pacific time on 14 April, almost all Airship services were unavailable. This occurred because our domain registration for urbanairship.com expired even though current evidence indicates that it was set up correctly to renew automatically. Airship engineers were made aware of the issue by internal monitoring when it began and immediately began working to restore services. After the critical impacts resolved, intermittent DNS errors continued to impact customers' ability to log into our go.urbanairship.com web dashboard (though go.airship.com and go.airship.eu were available) or use our go.urbanairship.com APIs in some locations for a further 24-48 hours. We have completed a Root Cause Analysis that is available via our Support team, are pursuing an investigation with our domain hosting provider to see why the auto-renew was not executed, and have identified investments to prevent any future recurrence of this magnitude. We sincerely apologize for any inconvenience this may have caused.

Posted Apr 16, 2020 - 13:35 PDT
Resolved
Based on the results of monitoring our services for the last 24 hours, we have determined that this incident has been resolved.

Please update any bookmarks to access the Airship Go dashboard from https://go.urbanairship.com to https://go.airship.com for the best results.

If you are still experiencing issues that you believe may be related to this incident please contact Airship Technical Support at https://support.airship.com
Posted Apr 16, 2020 - 10:27 PDT
Update
As of 8:30AM PT, 16:30 UTC, based on our worldwide monitoring, we believe that the DNS issues we were experiencing yesterday have been resolved.

We are continuing to monitor very closely. If you continue to see 503 Service Unvailable or DNS errors please contact Airship Technical Support at https://support.airship.com.

We will monitor this issue for the next 24 hours to ensure it is addressed for all customers and then will either resolve it or provide more updates should the status change.
Posted Apr 15, 2020 - 09:55 PDT
Update
We are continuing to monitor for any continuing issues. If you are impacted and using the API to send pushes, you may be able to use the Airship Dashboard to send pushes as an interim workaround.

Users who are unable to access the Airship Dashboard through https://go.urbanairship.com should use https://go.airship.com to access the Airship Dashboard instead.

If you are still experiencing errors while attempting to access Airship services please contact Airship Technical Support at https://support.airship.com
Posted Apr 15, 2020 - 07:46 PDT
Update
While the fix deployed at around 6:45 PST this morning was implemented successfully, the changes are still propagating to some regions. Users in impacted regions may experience intermittent errors accessing the Dashboard and APIs as changes propagate.

If you are impacted and using the API to send pushes, you may be able to use the Airship Dashboard to send pushes as an interim workaround.

Users who are unable to access the Airship Dashboard through https://go.urbanairship.com should use https://go.airship.com to access the Airship Dashboard instead.

While our engineering teams are continuing to work toward expediting a solution for those who are still impacted, we anticipate monitoring to continue through tomorrow morning at around 7:00 AM PST.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com.
Posted Apr 14, 2020 - 21:01 PDT
Update
While the fix deployed at around 6:45 PST this morning was implemented successfully, some users in select regions may experience intermittent errors accessing the Dashboard and APIs as changes propagate.

Users who are unable to access the Airship Dashboard through https://go.urbanairship.com should use https://go.airship.com instead.

We currently anticipate monitoring to continue through tomorrow morning at around 7:00 AM PST.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 15:01 PDT
Update
While the fix deployed at around 6:45 PST this morning was implemented successfully, some users in select regions may experience intermittent errors accessing the Dashboard and APIs as changes propagate.

Our engineering teams are working with the highest urgency to address any issues we encounter while monitoring the fix.

We currently anticipate monitoring to continue through tomorrow morning at around 7:00 AM PST.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 13:59 PDT
Update
We are continuing to monitor for any further issues.

Most Airship services have restored functionality, however, some users in select regions may continue to experience intermittent errors accessing the Dashboard and APIs as changes propagate. We will continue to update on the recovery as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 13:02 PDT
Update
We are continuing to monitor for any further issues.

Most Airship services have restored functionality, however, some users in select regions may continue to experience intermittent errors accessing the Dashboard and APIs as changes propagate. We will continue to update on the recovery as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 11:17 PDT
Update
Most Airship services have restored functionality, however, some users in select regions may continue to experience intermittent errors accessing the Dashboard and APIs as changes propagate. We will continue to update on the recovery as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 09:50 PDT
Update
Most Airship services have restored functionality, with some areas still experiencing delays. We are currently seeing degraded performance for the Airship Dashboard in select regions as well as delays in scheduled push notifications, reporting numbers, and automation triggers. Do not attempt to resend notifications to avoid sending duplicates. We will continue to update on the recovery as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 08:51 PDT
Update
We continue to see improvements to all Airship Services. Access to the Go Dashboard and push APIs have been restored but are still experiencing degraded performance. We are currently experiencing delays with Push notification delivery and Channel Registration at this time. Do not attempt to resend notifications to avoid sending duplicates. We will continue to update on the recovery as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 08:12 PDT
Update
We are seeing small improvements to the Airship dashboard at this time, however, all services are still experiencing issues. We will continue to update on the recovery as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 07:34 PDT
Monitoring
We have implemented a fix to restore access to the Airship dashboard and APIs. At this time all services are experiencing issues. We will continue to update on the recovery as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 06:45 PDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 14, 2020 - 06:03 PDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 14, 2020 - 04:57 PDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 14, 2020 - 04:10 PDT
Update
We are currently seeing frequent 5xx errors when attempting to access the go dashboard and several APIs including Push API, Named User, Channel Registration and Wallet APIs. We are still investigating the root cause of this issue.
Posted Apr 14, 2020 - 03:36 PDT
Identified
We’ve identified an issue that is returning frequent 5xx errors when accessing the Go Dashboard as well as the Channel Registration and Named user APIs. Push notifications through the dashboard are currently not available. Push API and previously scheduled push notifications will continue to deliver as expected.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 02:55 PDT
Investigating
We are currently investigating an issue that is resulting in failures when accessing the Airship Go Dashboard and certain APIs.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Apr 14, 2020 - 02:34 PDT
This incident affected: Web Dashboard, APIs, Mobile Messaging, Web Notifications, Mobile Wallet Passes, Email Messaging, SMS Messaging, Message Center Messaging, Real-Time Automation, Targeting & Personalization and Analytics and Data Solutions (Engagement Reporting, Real-Time Data Stream).
← Current Status Powered by Statuspage