Named User association, tags, and attributes delayed by an hour or more
Incident Report for Airship
Postmortem

Between approximately 8:37 AM and 12:01 PM US Pacific time on 8 March, named user changes and named user tag changes on our US cloud site may have been delayed in excess of 1 hour, up to a maximum of 2.5 hours. We also experienced some short periods of elevated 5xx errors from named user API endpoints as the system recovered towards the end of this period. While this should not have impacted message delivery directly, it's possible that audience lookups for sends may have utilized data that were not yet up to date. This issue was due to elevated levels of a particular kind of traffic, and was resolved by blocking that traffic and reaching out to the customers involved to help them use our platform more efficiently. Our Engineering team is investigating changes we can make to the system to handle this sort of traffic in large volumes better in the future, preventing future recurrences. We apologize for any inconvenience.

Posted Mar 09, 2021 - 11:51 PST

Resolved
We have resolved the issue which has resulted in delays to named user associations, tags, and attributes.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Mar 08, 2021 - 14:03 PST
Monitoring
We have implemented a change to mitigate the delays in our named user service. Changes in named user associations, tags, and attributes are now occurring as expected. We are continuing to monitor and investigate the issue and its cause.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Mar 08, 2021 - 12:36 PST
Investigating
We have identified an issue causing changes in named user associations, tags, and attributes to be delayed for greater than one hour. Our engineers are actively working on resolving this issue.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Mar 08, 2021 - 10:49 PST
This incident affected: Targeting & Personalization.