Reporting Delays for Device Events
Incident Report for Airship
Postmortem

Between approximately 11:30am and 2:25pm US Pacific time on 17 March, an issue with changing over a data source that counts message sends and campaign events caused some of these reports to be moderately delayed. There may also be incomplete or missing data for sends from this time period. Please feel free to reach out to our support team at http://support.airship.com if you need assistance accessing your data. We apologize for any inconvenience.

Posted Mar 27, 2020 - 08:18 PDT

Resolved
This incident has been resolved. We have achieved significant improvements to the reporting delays, and delays to Unique/Influenced Opens are within normal parameters. Inactivity Automations have been re-enabled at this time. All other notifications, messages, and automation continue to send normally.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Mar 25, 2020 - 09:51 PDT
Update
We are continuing to monitor the progress of the fix currently in place, and are seeing significant improvements. There are still some reporting delays, particularly with opens (including influenced opens) and device-side custom events. Automations based on region events and device-side custom events remain delayed. Inactivity automations are still paused at this time to avoid sending in error. All other notifications, messages, and automation continue to send normally.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Mar 25, 2020 - 08:46 PDT
Monitoring
A fix has been implemented to resolve this issue, and we are currently monitoring the results. At this time, we are still seeing significant reporting delays, particularly with opens (including influenced opens), and device-side custom events. These are also delayed in Real-Time Data Streaming and Performance Analytics. Automations based on region events and device-side custom events are also still impacted. Inactivity Automations are still paused in order to prevent sends erroneously. All other notifications, messages, and automation continue to send normally.

If you have any questions or concerns please contact Airship Technical Support at https://support.airship.com
Posted Mar 25, 2020 - 07:05 PDT
Update
We are currently experiencing an issue which is resulting in multiple hours of reporting delays, particularly with opens (including influenced opens), and device-side custom events. These are also delayed in Real-Time Data Streaming and Performance Analytics. Automations based on region events and device-side custom events are also delayed. Inactivity automations have also been paused for the duration of the incident so they are not sent in error. All other notifications, messages, and automations should be sending normally
Posted Mar 24, 2020 - 22:06 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2020 - 22:05 PDT
Update
Our engineers are continuing to work with a third party provider to resolve this issue. Send counts should now be up to date, but there are still significant delays with opens (including influenced opens) and device-side custom events. Automations based on region events and device-side custom events are also delayed. Inactivity automations have been paused for the duration of the incident so they are not sent in error. All other notifications, messages, and automations should be sending normally.
Posted Mar 24, 2020 - 21:31 PDT
Update
Our engineers are continuing to work to resolve this issue. Current status is we are seeing multiple hours of reporting delays, particularly with sends, opens (including influenced opens), and device-side custom events. Automations based on region events and device-side custom events are also delayed. Inactivity automations have also been paused for the duration of the incident so they are not sent in error. All other notifications, messages, and automations should be sending normally.
Posted Mar 24, 2020 - 17:55 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2020 - 16:05 PDT
Update
We are continuing to work on a fix for this issue. Delays to reporting and Automation triggers are expected at this time.
Posted Mar 24, 2020 - 15:02 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2020 - 14:01 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2020 - 12:52 PDT
Update
We are continuing to work on a fix for this issue. Delays to reporting and Automation triggers are expected at this time.
Posted Mar 24, 2020 - 11:38 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2020 - 10:14 PDT
Identified
We’ve identified the root cause of this issue and our engineers are actively working to resolve the issue. At this time Reporting data, Device Properties and Device Events are still delayed.

If you have any questions or concerns please contact Airship Support at support.airship.com
Posted Mar 24, 2020 - 09:20 PDT
Investigating
We are currently investigating an issue that is resulting in delays in event reporting for devices. Events that are used for Automations are currently delayed by an hour or more. Some device property data updates (such as location history and certain device property tags) are also delayed/impacted. All other services, including notification delivery, are working as expected at this time.

If you have any questions or concerns please contact Airship Support at support.airship.com
Posted Mar 24, 2020 - 08:50 PDT
This incident affected: Analytics and Data Solutions (Data Collection & Processing, Engagement Reporting, Real-Time Data Stream, Performance Analytics) and Real-Time Automation.