Increased rate of 5XX errors in API and Airship Dashboard
Incident Report for Airship
Postmortem

Between approximately 7:03 PM on 9 February and 9:04 AM on 10 February US Pacific time, our systems experienced an issue resulting in intermittent elevated levels of 5xx errors on our APIs, particularly named user API. This may have also impacted contact lookup functionality in the go dashboard. Between 7:15 AM and 9:04 AM on 10 February US Pacific time, it significantly impacted our push APIs, and caused some sending delays as that system recovered around 9:00AM. We also saw processing delays in excess of 1 hour on some device registration activities between 8:48 and 9:33 AM on 10 February US Pacific time. These issues impacted customers hosted on our US cloud site. The EU site was not impacted. It should also not have impacted messaging prepared ahead of time through automations or the scheduling system. Our team was able to identify that the issue was due to network communication failures between internal services and our central database, though a root cause analysis is still being performed.

Posted Feb 12, 2021 - 08:48 PST

Resolved
We have resolved the issue that resulted in intermittent errors with contact lookup in the Airship Dashboard and some API endpoints (including push API) and registration delays.

All systems should be fully operational.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Feb 10, 2021 - 10:19 PST
Monitoring
A fix has been implemented. We are currently monitoring the results.

We will update our status page as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Feb 10, 2021 - 09:20 PST
Update
In addition to intermittent errors with contact lookup in the Airship Dashboard and some API endpoints, including push API, customers may experience registration delays.

If you are encountering any of the above issues, please retry your requests

Our engineering teams are working to resolve this issue. We will update our status page as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Feb 10, 2021 - 08:55 PST
Identified
We are continuing to see intermittent errors with contact lookup in the go dashboard and some API endpoints.

Our engineering teams are working to resolve this issue. We will update our status page as soon as we have more information.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Feb 10, 2021 - 08:28 PST
This incident affected: Web Dashboard and APIs.