Blank

Umbraco Status page


Welcome to the Umbraco status page. On this page, you can see the current operational status as well as plans for scheduled maintenance and automatic upgrades for all our cloud offerings: Umbraco Cloud and Heartcore.

Subscribe to updates above to get the latest status send straight to your inbox.

If you’re experience issues with your cloud project which does not seem to relate to the current operational status, please go to Our Umbraco and search for the issue or reach out to the Umbraco Support in the portal chat.

Resolved: Umbraco Cloud incident 22/10/2020
Incident Report for Umbraco Cloud
Postmortem

During the period of Thursday 22nd of October at 07:10 - 15:55 CEST, Umbraco Cloud suffered an issue with our hosting management system, which caused downtime and/or general instability for several hosted websites on the platform.

This post mortem will outline the events as they occurred, give details on the root cause and explain what steps we are taking to improve and prevent similar incidents based on these learnings.

Summary and root cause

On October 22nd, 2020 at 07:10 CEST websites on Umbraco Cloud started to become unavailable with timeout errors due to an issue with a central component of our hosting management system.

We were able to execute an emergency patch at 07:49 AM CEST, and a full recovery for all sites was initiated immediately.

Recovery timeline 

Site recovery timeline:

65% of sites recovered before 09:00 CEST

95% of sites recovered before 12:30 CEST

100% of sites recovered before 15:55 CEST

Follow Up

Additional root cause analysis was completed Friday, October 23rd and improvements to infrastructure and internal processes were made. This includes more thorough monitoring and prioritisation of fixes related to this issue in our hosting platform which we are now working on implementing in order to prevent future related incidents from happening as well as shortening the lead time from detection to deployment of critical patches if necessary. 

We apologize for the inconveniences this incident has caused you and your customers. We want to underline that our work to ensure stable operations on Umbraco Cloud does not end with this update but is ongoing and is of the highest priority to us.

CTO - Jacob Midtgaard-Olesen

Umbraco HQ

Posted Oct 26, 2020 - 10:28 CET

Resolved
This incident has been resolved.

We apologize for the inconvenience this incident has caused. We are now moving on to investigating the cause of the incident and will release a postmortem about this here on the Umbraco Cloud Status Page no later than Monday, October 26th at 10:00 CET.
Posted Oct 22, 2020 - 17:25 CEST
Monitoring
All sites are up and running. We are currently monitoring and expecting to close the incident within the next hour.

We will update again no later than 4 PM UTC/18:00 CEST or when the incident has been resolved.
Posted Oct 22, 2020 - 16:36 CEST
Update
We continue to be in “Minor” impact mode as more sites are now back to normal operations.

Given that the vast majority of sites are now up and running, we’ll extend our regular updates here on Status Page to every two hours.

We'll provide an update on this incident no later than at 3 PM UTC/ 17:00 CEST.
Posted Oct 22, 2020 - 14:55 CEST
Update
We are continuing to seeing fewer sites affected by this incident and are now able to turn the impact down to “Minor”.

We are no longer experiencing issues with creating new projects.

We'll provide an update on this incident here no later than at 2 PM UTC.
Posted Oct 22, 2020 - 13:56 CEST
Update
We're still seeing that a majority of sites are up and running. Recovery continues for a minority of sites and we’re working hard to also get these back to normal operations.

You may still experience issues when creating a new project whether your site is available or not - we are investigating this further.

We'll provide an update on this incident here no later than at 1 PM UTC.
Posted Oct 22, 2020 - 13:00 CEST
Update
We are continuing to seeing a reduction of sites being affected and a majority of sites are currently experiencing normal operations.

Some sites remain either unavailable or slow and may experience issues when creating new projects.

We'll provide an update on this incident here no later than at 12 PM UTC.
Posted Oct 22, 2020 - 11:58 CEST
Update
We are still recovering sites.

We are seeing a reduction of sites being affected, however, some sites remain either unavailable or slow and you may experience issues when creating new projects.

We'll provide an update on this incident here no later than at 11 AM UTC.
Posted Oct 22, 2020 - 10:59 CEST
Identified
We have now identified the issue and are recovering sites in order to get operations back to normal.

Several sites remain either unavailable or slow and you may experience issues when creating new projects.

We'll provide an update on this incident here no later than at 9 AM UTC.
Posted Oct 22, 2020 - 10:02 CEST
Investigating
We are currently investigating this issue.
Posted Oct 22, 2020 - 07:39 CEST