Several Chicago hosts experienced simultaneous disk failures causing significantly degraded states impacting application performance. These failures additionally prevented some existing apps on the affected hosts from stopping properly, causing deployments for these apps to get stuck in a 'pending' state while their hosts were unresponsive.
The issue was eventually resolved by identifying and repairing the bad disks on all affected hosts, and we will be making improvements to our deployment system to prevent similar stuck-deployment issues from occurring in the future.
Posted May 09, 2022 - 18:05 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 09, 2022 - 08:00 UTC
Identified
It definitely looks like a hardware issue! We're pulling the affected host out of the rotation and are continuing to monitor.
Posted May 09, 2022 - 03:10 UTC
Investigating
This is an issue with a specific host; it may be hardware-related. We're looking in to it.
Posted May 09, 2022 - 02:54 UTC
This incident affected: Regional Availability (ORD - Chicago, Illinois (US)).