High levels of errors from our named user and channel APIs
Incident Report for Airship
Postmortem

Between approximately 1:25 AM and 1:04 PM US Pacific time on 24 May, we experienced an issue with named user and channel registration updates, listings, and lookups on our US cloud site. This manifested as elevated levels of 5xx error responses from named user and channels APIs between approximately 1:25 and 10:19 AM, and delays in registration or named user processing in excess of one hour between 3:57 AM and 1:04 PM. Our engineering team is putting together a plan to enhance our API limits around these systems to prevent similar issues in the future, as well as provide better, more automatic scaling. We apologize for any inconvenience.

Posted May 26, 2021 - 10:08 PDT

Resolved
This issue has been resolved. Registrations are being processed as expected in our system.

If you have any questions or concerns please contact Airship Technical Support at support.airship.com
Posted May 24, 2021 - 13:47 PDT
Monitoring
We have seen significant improvements from the fix that was implemented earlier. Some customers may continue to see delayed registrations, but the system is recovering and we are continuing to monitor the results.

We will continue to update this status page as we have additional information. If you have any questions or concerns please contact Airship Technical Support at support.airship.com.
Posted May 24, 2021 - 11:54 PDT
Identified
Our engineers have identified the issue and have implemented a potential fix. We are currently monitoring the results.

We will continue to update this status page as we have additional information. If you have any questions or concerns please contact Airship Technical Support at support.airship.com.
Posted May 24, 2021 - 10:38 PDT
Update
We are continuing to investigate an issue that is resulting in high levels of errors from our named user and channel APIs on our US site. Customers may be unable to look up contacts and channels in the Airship Dashboard, and may notice delays of over an hour for named user updates.

Sends do not appear to be impacted by this issue.

Our engineering team is continuing to work toward identifying and resolving this issue, and we continue to see improvements.

We will update this status page as soon as we have additional information. If you have any questions or concerns please contact Airship Technical Support at support.airship.com.
Posted May 24, 2021 - 10:22 PDT
Investigating
We are currently investigating an issue resulting in high levels of errors from our named user and channel APIs on our US site since ~1:25 PST. This is also impacting the ability to look up contacts and channels in the Go dashboard. Customers may also have noticed delays of over one hour for named user updates. Sendings don’t seem to be impacted by the issue.

Our engineering team is working to identify and resolve the issue, and is seeing significant improvements. We will update this status page as soon as we have more information.

Please feel free to contact Airship Technical Support at https://support.airship.com if you have any questions or concerns.
Posted May 24, 2021 - 06:09 PDT
This incident affected: APIs.