Blank
Umbraco Status page
Umbraco Status page
Welcome to the Umbraco status page. On this page, you can see the current operational status as well as plans for scheduled maintenance and automatic upgrades for all our cloud offerings: Umbraco Cloud and Heartcore.
Subscribe to updates above to get the latest status send straight to your inbox.
If you’re experience issues with your cloud project which does not seem to relate to the current operational status, please go to Our Umbraco and search for the issue or reach out to the Umbraco Support in the portal chat.
During the period of Thursday 12th of November at 06:30 AM - 11:30 AM CET, Umbraco Cloud suffered an issue with our hosting management system which caused downtime and/or general instability for several hosted websites on the platform.
This post mortem will outline the events as they occurred, give details on the root cause and explain what steps we are taking to improve and prevent similar incidents based on these learnings.
Summary and root cause
Similar to previous and recent incidents, the issue was related to the hosting management platform of Umbraco Cloud and thus affected most customers.
The issue of our hosting management platform was identified to be performing insufficiently from a database central to the orchestration of website distribution on the hosting platform. An issue that was mitigated during planned maintenance on November 13th.
For the incident on November 12th, as a consequence of the hosting management performance issue, a recovery of our hosting platform was automatically triggered, causing a majority of websites to become unavailable or unresponsive while the recovery was in progress.
Recovery timeline
Site recovery timeline:
25 % of sites recovered before 06:45 am CET
50% of sites recovered before 07:00 am CET
80% of sites recovered before 08:00 am CET
90% of sites recovered before 08:30 am CET
99% of sites recovered before 10:00 am CET
100% of sites recovered before 11:00 am CET
Follow Up
Emergency maintenance was scheduled for November 14th at 01:00 CET, but due to issues on November 13th starting at 19:59 CET, the maintenance was escalated and started November 13th at 23:30 CET.
Additional maintenance is planned at midnight, 00:00 CET between Friday, November 20th and Saturday, November 21st. and ends 8 hours later at 08:00 CET November 21st. During this maintenance the database in question will be replaced, along with additional improvements, to ensure continued stability of Umbraco Cloud.
Furthermore, we are in the process of strengthening Umbraco Cloud with additional new infrastructure and services in the near future, including:
- Migration to Azure Blob storage (ongoing)
- Migration to Cloudflare SSL for SaaS, to replace Umbraco Latch
- Migration to Azure App Services to replace current hosting management platform
Live webinars for Cloud customers
We’re hosting a live post-mortem webinar today, Wednesday November 18th. The webinar will give you a thorough overview of the work we have planned going forward to prevent incidents like the once we’ve experienced lately. We’re hosting the webinar twice so you can pick a time that suits you:
- Wednesday November 18th at 10:00 CET: https://umbraco.zoom.us/webinar/register/WN_up_InRstR8qv8y0Lx_XdcQ
- Wednesday November 18th at 17:00 CET: https://umbraco.zoom.us/webinar/register/WN_JjZIVZTtTuS7eY-h5XcBLA
We need to do better.
As stated on the released management statement on the Umbraco blog (https://umbraco.com/blog/umbraco-cloud-incident-1311-14112020/) - we are painfully aware that the Umbraco Cloud incidents within the last weeks have been affecting you as our customer. We do apologize and please convey our apologies to your customers.
We know that we need to do better. The live webinars above will give you a thorough walk-through of our plans now and going forward in order to secure performance stability on Umbraco Cloud and thus, hopefully, restore trust in our service again. We advise all interested Umbraco Cloud customers to sign up for the live webinars.
Within the next 14 days, a blog post will be posted on the Umbraco blog (https://umbraco.com/blog/) with more details about the development and future plans for the Umbraco Cloud platform. Update: the blog post will be posted in the week commencing December 7th. Update: the blog post will be posted on the Umbraco Blog, Monday, December 14th.
Kim Sneum Madsen, Umbraco CEO:
I sincerely apologize for the inconvenience this incident has caused both for you and your clients.
We are now moving on with the further investigation and will release a post-mortem about this incident here on the Umbraco Cloud Status Page no later than on Wednesday 18th of November at 10 CET.
In addition to the post-mortem, we’re hosting two post-mortem webinars.
The webinar will give you a thorough overview of the work we have planned going forward to prevent incidents like these. We’re hosting the webinar twice so you can pick a time that suits you.
*Sign up for the post-mortem webinars*
- Wednesday November 18th at 10:00 CET: https://umbraco.zoom.us/webinar/register/WN_up_InRstR8qv8y0Lx_XdcQ
- Wednesday November 18th at 17:00 CET: https://umbraco.zoom.us/webinar/register/WN_JjZIVZTtTuS7eY-h5XcBLA
If you have subscribed to updates here on the Status Page, you will be notified by email as soon as the post-mortem is ready on Wednesday.
*Scheduled maintenance Saturday 14th of November at 1 AM - 3 AM CET*
We already have several actions in place to improve the situation. One of them will take place on Saturday, 14th of November at 1 AM - 3 AM CET where we have announced scheduled maintenance for Umbraco Cloud (see the annoncement on https://status.umbraco.io/). We don’t expect any downtime for sites during the maintenance. More information about the actions we are taking to prevent incidents will be shared in the webinar.
Again, I give you my deepest apologies and hope you will extend my apologies to your clients.
Know that I along with the entire team at Umbraco are committed to stabilizing operations and that there is no doubt in my mind that this is the most important task we have if we are to run a sustainable business.
Kim Sneum Madsen, Umbraco CEO.
We will keep monitoring.
Next update will be no later than 4 PM CET
We will update with a status no later than at 2 PM CET.
We will update again no later than 12 AM CET.
We will update with a status no later than at 10 AM CET.
We will update with a status no later than 9 AM CET.