Wallet Pass Updates Delayed
Incident Report for Airship
Postmortem

12:24 AM 5 May and 5:33 AM 8 May as well as between 9:21 PM 8 May and 4:20 PM 9 May US Pacific time, Wallet pass updates may have been delayed over one hour. This issue did not impact new pass creations and downloads, updates via segment, or scheduled updates, or any of our non-Wallet push notification systems. However, customers may have also noticed some brief periods of elevated error response rates from Wallet API endpoints during these times. This issue was caused by increased workload on this system, which our Engineering team has ameliorated with queueing and database improvements. They are also in the process of improving alerting and monitoring for this system to help us respond more quickly to similar issues in the future. We apologize for any inconvenience.

Posted May 11, 2021 - 11:33 PDT

Resolved
This issue has been resolved. All wallet pass update services are working as expected. We are continuing to monitor this behavior internally for any changes.

All services within Airship are now working as expected.

If you have any additional questions or concerns, please contact Airship Technical Support at support@airship.com.
Posted May 08, 2021 - 08:16 PDT
Identified
We have identified the issue that is resulting in delays to pass updates. Our system is currently processing pass updates and showing some signs of improvement. We expect to provide the next update tomorrow morning no later than 9:00 AM PST.

At this time, all other services through both the dashboard and the API are working as expected.

If you have any additional questions or concerns, please contact Airship Technical Support at support@airship.com.
Posted May 07, 2021 - 16:38 PDT
Update
We are continuing to investigate an issue that is causing a delay in processing updates for wallet passes. New pass creations/installs and scheduled/segmented pass updates are unaffected.

All other services through both the dashboard and the API are working as expected.

Our engineering teams are working to resolve this issue. We will update our status page as soon as we have more information.

If you have any additional questions or concerns, please contact Airship Technical Support at support@airship.com.
Posted May 06, 2021 - 22:03 PDT
Update
We are continuing to investigate an issue that is causing a delay in processing updates for wallet passes. New pass creations/installs and scheduled/segmented pass updates are unaffected.

All other services through both the dashboard and the API are working as expected.

Our engineering teams are working to resolve this issue. We will update our status page as soon as we have more information.

If you have any additional questions or concerns, please contact Airship Technical Support at support@airship.com.
Posted May 06, 2021 - 18:08 PDT
Investigating
We're currently investigating an issue that is causing a delay in processing updates for wallet passes. New pass creations/installs and scheduled/segmented pass updates are unaffected.

All other services through both the dashboard and the API are working as expected.

Our engineering teams are working to resolve this issue. We will update our status page as soon as we have more information.

If you have any additional questions or concerns, please contact Airship Technical Support at support@airship.com.
Posted May 06, 2021 - 13:55 PDT
This incident affected: Mobile Wallet Passes.