Shared Infrastructure

All tasks enqueued during the last few hours have been concluded. All new tasks are being executed normally. We are taking steps to prevent administrative tasks to cause such interference again.

The system is normalizing itself and Tasks triggered in the last few hours are currently running.

We are aware that Asynchronous Tasks (which include, for example, Datapoint Imports) are taking a long time to run. This is due to the increased load in our Asynchronous Tasks system caused by internal administrative tasks.

API is back to normal. Resolved itself after those requests have been treated.

Our system experiences some slowness due to a high number of requests.

We confirm that the all issues are over. The app is now reliably available on Asia, North America, and Australia. Live updates (on all regions) are also working properly.

Our networking provider, Cloudflare, reported that they are experiencing difficulties at the moment. Those difficulties match up with the issues we are experiencing. No action can be performed on our part to alleviate those issues. A detailed report can be found here: https://www.thousandeyes.com/blog/cloudflare-outage-analysis-september-16-2024

We are aware that our live updates system is also unavailable across all regions. This affects the web and mobile applications, which will fail to live update when other users perform changes.

We confirm that the outage is localized to those regions and does not affect Europe.

We are aware that the application is unavailable on Europe, Asia, and North America regions. This affects the API and our Web and Mobile application on those regions. We are investigating.

The usage of the database is back to normal. We are still monitoring and investigating about what caused this incident.

We're experiencing an issue that put the API down. We identified a peak of usage of our database. We are making actions to resolve the issue and investigate.

The bug regarding Routines has been fixed, but unfortunately the correct data from that period cannot be recovered. We will assist the recovery of our customers' operations in the following way: - Providing a list of the affected Routine Executions that have invalid data. - Offering to remove the affected Routine Executions. - Rebuilding the data on Dynamic KPIs that are based on those Routines.

Due to a regression introduced by a change in the application, from Wednesday 2024-08-21 01:04 PM UTC to Thursday 2024-08-22 8:24 AM UTC, any usage of Routines resulted in invalid data being recorded. This was due to a bug that displayed incorrect Routine Steps when starting, submitting, or editing a Routine Execution; and when editing the Steps of Routines. When used during that time period, Routines were missing some Steps, or had Steps that were displayed or recorded data in an unexpected way. Therefore, the data on those Routine Executions is invalid and cannot be exploited. Dynamic KPIs based on those Routine Executions will also display incorrect data. The issue affected the Webapp, the Mobile Application, and the ability to retrieve data from Routines on our public API.

The root cause comes from a feature that was supposed to improve the loading time of images and avatars. This feature was intended to enhance the way thumbnails are managed in the legacy code. This improvement has been postponed and will be implemented in the new stack.

Service on the API is restored. We are investigating the cause of the incident.

We're experiencing an issue that put the API down. We are making actions to resolve the issue and investigate.