This incident is now resolved.
The issue affected a small number of projects during automatic upgrades, where an upgrade could become stuck without completing. Around 2.7% of attempted upgrades were affected, and some of these completed successfully despite showing as stuck.
We have added new monitoring to detect this behaviour earlier, and a fix is being rolled out ahead of the next round of automatic upgrades.
If you believe your environment was affected, or an upgrade appears stuck in future, please reach out to our support team and we will resolve it for you.
We apologise for any inconvenience caused.
Resolved
This incident is now resolved.
The issue affected a small number of projects during automatic upgrades, where an upgrade could become stuck without completing. Around 2.7% of attempted upgrades were affected, and some of these completed successfully despite showing as stuck.
We have added new monitoring to detect this behaviour earlier, and a fix is being rolled out ahead of the next round of automatic upgrades.
If you believe your environment was affected, or an upgrade appears stuck in future, please reach out to our support team and we will resolve it for you.
We apologise for any inconvenience caused.
Identified
We have identified the cause of the increased number of hanging automatic upgrades. An issue in our upgrade orchestration can cause an upgrade to become stuck without receiving a completion status. Affected upgrades do not fail, but their activity may continue running and the environment can report as busy for other operations.
As a short-term measure, our support team can cancel stuck upgrade activities to unblock affected environments. If your environment appears busy or an upgrade seems stuck, please reach out to support and we will resolve it for you.
A fix that handles these cases more gracefully is ready and is expected to be rolled out within the coming week. We will post a further update once it is deployed.
Investigating
Some customers may experience hanging upgrades on their Umbraco Cloud projects.