Blank
Umbraco Status page
Umbraco Status page
Welcome to the Umbraco status page. On this page, you can see the current operational status as well as plans for scheduled maintenance and automatic upgrades for all our cloud offerings: Umbraco Cloud and Heartcore.
Subscribe to updates above to get the latest status send straight to your inbox.
If you’re experience issues with your cloud project which does not seem to relate to the current operational status, please go to Our Umbraco and search for the issue or reach out to the Umbraco Support in the portal chat.
During the period of Friday 13th of November at 20:00 - 22:30 CET, Umbraco Cloud suffered an issue with our hosting management system, which caused downtime and/or general instability for several hosted websites on the platform.
As a consequence of these issues and previous related incidents, emergency maintenance was initiated November 13th 23:30.
This post mortem will outline the events as they occurred, give details on the root cause and explain what steps we are taking to improve and prevent similar incidents based on these learnings.
Summary and root cause
Similar to previous and recent incidents, the issue was related to the hosting management platform of Umbraco Cloud and thus affected most customers.
The root cause had already been identified during the incident on November 12th as an insufficient performance from a database central to the orchestration of website distribution on the hosting platform. The already scheduled emergency maintenance was thus escalated to start as early as possible as part of this incident.
As a consequence of both incident and later emergency maintenance, several websites on Umbraco Cloud experienced downtime during this period.
For the initial incident, Umbraco Cloud was 50% recovered at 21:45 CET and 100% recovered before 22:30 CET.
For the following emergency maintenance, about 50% of our customers experienced additional negative impact on hosting until maintenance was successfully completed and the system had fully recovered.
Follow Up
The maintenance successfully mitigated the immediate issue with performance.
Additional maintenance is planned at midnight, 00:00 CET between Friday, November 20th and Saturday November 21st. and ends 8 hours later at 08:00 CET November 21st. During this maintenance the database in question will be replaced, along with additional improvements, to ensure continued stability of Umbraco Cloud.
Furthermore, we are in the process of strengthening Umbraco Cloud with additional new infrastructure and services in the near future, including:
- Migration to Azure Blob storage (ongoing)
- Migration to Cloudflare SSL for SaaS, to replace Umbraco Latch
- Migration to Azure App Services to replace the current hosting management platform
We’re hosting a live post-mortem webinar today, Wednesday November 18th. The webinar will give you a thorough overview of the work we have planned going forward to prevent incidents like the once we’ve experienced lately:
- Wednesday November 18th at 17:00 CET: https://umbraco.zoom.us/webinar/register/WN_JjZIVZTtTuS7eY-h5XcBLA
We need to do better.
As stated on the released management statement on the Umbraco blog (https://umbraco.com/blog/umbraco-cloud-incident-1311-14112020/) - we are painfully aware that the Umbraco Cloud incidents within the last weeks have been affecting you as our customer. We do apologize and please convey our apologies to your customers.
We know that we need to do better. The live webinars above will give you a thorough walk-through of our plans now and going forward in order to secure performance stability on Umbraco Cloud and thus, hopefully restore trust in our service again. We advise all interested Umbraco Cloud customers to sign up for the live webinars.
Within the next 14 days, a blog post will be posted on the Umbraco blog ([https://umbraco.com/blog/](https://umbraco.com/blog)) with more details about the development and future plans for the Umbraco Cloud platform. Update: the blog post will be posted in the week commencing December 7th. Update: the blog post will be posted on the Umbraco Blog, Monday, December 14th.
Rest assured that we will keep taking all the necessary steps to prevent incidents from happening in the future. This is important to us, and we are pursuing multiple ways of ensuring stable operations.
Earlier today, we shared a Management Statement regarding the incident. You can read it right here: https://umbraco.com/blog/umbraco-cloud-incident-1311-14112020/
If you're experiencing any issues, please contact the Umbraco Cloud support at contact@umbraco.com, and they’ll be ready to help. Furthermore, a postmortem will be available within 5 workdays.
We sincerely apologize for the inconvenience caused.
Next update will be no later than at 13:30 CET.
Next update will be no later than at 13:30 CET.
Next update will be no later than at 11:30 CET.
Next update will be no later than at 10:30 CET.
Next update will be no later than at 09:30 CET.
Next update will be no later than at 08:30 CET
A subset of sites are not responding yet as a consequence of them having been offline for the maintenance operations, but the number of sites online is continuously increasing and we expect all sites to be back to normal operations soon.
We will not end the maintenance window before all sites are back online again.
Next update will be no later than 07:30 CET
Next update will be no later than at 06:30 CET
Next update will be no later than 05:30 CET
Next update will be no later than at 04:30 CET
Next update will be no later than at 03:30 CET
We will update here no later than 02:30 CET
Next update will be no later than at 01:30 CET
We expect additional downtime of sites during the maintenance.
We are painstakingly aware of how bad the current situation is for you and your customers.
We have external consultants on board and are in dialogue with Microsoft on the issue.
We are of course actively monitoring the situation closely and are ready to mitigate any negative impact during and after the maintenance.
After the maintenance we will know more and reassess which further actions might need to be taken.
We don’t have all the answers right now but rest assured that we will give full disclosure once we do and will keep you updated on statuspage as soon as we have new information to share.
Our sincere apologies
Kim Sneum Madsen - CEO
Next update will be no later than at 23:45 CET.
Next update will be no later than at 23:30 CET.
Next update will be no later than at 22:30 CET.