Our platform experienced several issues today due to problems experienced by our third party cloud provider.
First, between about 2:46-3:00AM US Pacific time, our push API was returning a high number of errors, making it difficult to send or schedule push messages. Our web dashboard also returned a variety of errors during this period.
By about 3AM, push API and critical website functionality had improved. Between about 3:00 and 3:30AM US Pacific time, there were significant delays in delivering messages and several other APIs continued to return a large number of errors.
Finally, between about 3:30 and 10:30AM Pacific, all other issues had resolved except for a few device registration/lookup APIs and systems still exhibiting significant problems or delays. This also impacted the ability of customers to modify test lists.
Around 10:30AM, we were able to implement a final workaround for remaining issues still being experienced by our third party cloud provider. We sincerely apologize for the inconvenience, and we are both awaiting an RCA from our provider and working internally on improving our methods for troubleshooting, communication, and developing workarounds for similar issues in the future.