Named User API Delays
Incident Report for Airship
Postmortem

Between approximately 4:00 AM and 10:00 AM US Pacific time on 15 December, customers on our US cloud site may have experienced an elevated level of error responses from our various Named User API endpoints. As such, unless those requests were successfully retried, this may have prevented our systems from getting updated information about devices and sending messages to them if they were targeted by named user or named user tags. The issue was resolved by adding more compute resources to the service. Later on, our Backend Engineering team discovered that a significantly large amount of CPU on named user API instances was being used up by unnecessary logging infrastructure, which they then removed. This will prevent future recurrences of the same service defect. Engineering is also planning to improve monitoring and alerting for this API so they will become aware of similar problems and resolve them more quickly in the future.

Posted Dec 22, 2020 - 07:44 PST

Resolved
We have addressed the issue that was resulting in an increased rate of 5xx errors when using our Named User APIs.

All services are now functioning normally.

If you have any questions or concerns, please contact Airship Technical Support at https://support.airship.com/
Posted Dec 15, 2020 - 12:39 PST
Monitoring
Our engineering teams have implemented a solution for this issue and we are currently monitoring the results.

If you have any questions or concerns, please contact Airship Technical Support at https://support.airship.com/
Posted Dec 15, 2020 - 10:17 PST
Investigating
We are currently investigating an issue that is resulting in an increased rate of 5xx errors when using our Named User API.

All other services are functioning normally.

Our engineering teams are working to resolve this issue. We will update our status page as soon as we have more information.

If you have any questions or concerns, please contact Airship Technical Support at https://support.airship.com/
Posted Dec 15, 2020 - 09:30 PST
This incident affected: APIs.