Post-Mortem Report: Umbraco Cloud Outage – June 12, 2025

On June 12, 2025, Umbraco Cloud experienced a service disruption affecting several parts of our platform. We want to share a transparent overview of what happened, what caused it, and how we’re taking action to reduce the risk of similar incidents in the future.

What Happened

Between 20:08 and 23:18 CEST, Umbraco Cloud Websites, Cloud & Heartcore services became unresponsive and returned 500 Internal Server Errors. Affected systems included:

• Umbraco Cloud Websites internal hostnames ({project}.{region}.umbraco.io)
• Umbraco Cloud Portal
• Heartcore REST and GraphQL APIs
  

Root Cause

Several Umbraco Cloud & Heartcore components depend on Cloudflare KV for normal operations, this upstream failure caused our services to return 500 errors.

A failure in Cloudflare Workers KV (Key Value) Store - which we use for routing, configuration, and other operational needs - caused read/write operations to stop functioning. The root of this failure was in Cloudflare’s underlying storage infrastructure. As a result, key services in Umbraco Cloud could no longer complete requests and returned errors instead.

Trust in Cloudflare & Managing Risk

Cloudflare is a key dependency in our architecture, and we rely on their commitment to a 100% SLA-backed uptime. In their public incident report, Cloudflare acknowledged the architectural weakness that led to the outage and shared the work already underway to reduce such risks in the future. This includes strengthening their infrastructure and minimizing reliance on any single provider.

We trust Cloudflare to follow through on these commitments and continue to invest in the resilience of their platform.

Steps Forward & Lessons Learned

This incident highlights the need to design systems that can tolerate failures—even in services considered highly reliable.

What We’re Doing

• We will evaluate alternative request routing strategies and graceful handling of errors in our Cloudflare deployments that would lessen the impact of Cloudflare KV outages in the future. 

  • Exploring whether critical configuration data can be mirrored
  • Exploring reducing the reliance on real-time KV access during its outage.

• Investigating graceful degradation mechanisms to maintain partial service availability during similar events.

Lessons Learned

• Even highly available platforms can be impacted by upstream single points of failure.
• Core functions like routing and API access should include fallback mechanisms where possible.
• Our existing monitoring and incident response processes worked as intended, enabling us to detect, escalate, and track the incident effectively.
  

Final Thoughts

We understand that any disruption in service can be frustrating. We sincerely apologize for the inconvenience caused and thank you for your continued trust and patience. We're committed to improving the resilience and reliability of Umbraco Cloud and will continue to share updates as we strengthen our systems and processes.

— The Umbraco Cloud Team