Resolved
All MPG clusters are back to full, normal operations.
Monitoring
All MPG clusters are reachable.
Identified
We are still continuing cleanup on some clusters.
Identified
All cluster primary and pgBouncer machines are now healthy and operating normally.
We are still continuing cleanup on some clusters with lagging or degraded replicas, but this should not impact writes or reads to clusters.
Identified
We are continuing to work on restoring all clusters to full health.
Identified
With the underlying incident stabilizing (https://status.flyio.net/incidents/3npj6935byt4) we are seeing improvements amongst impacted clusters. We continue to work on restoring all clusters to full health.
Identified
A number of clusters in IAD, AMS, and SIN regions continue to see degraded replicas and PGBouncers at this time. A smaller number of clusters in these regions are also seeing disruption to their primaries. We continue to work on restoring full cluster health in all regions.
Identified
A small number of MPG clusters in the AMS and IAD region are currently in degraded states due to downstream impact from this Machines API issue: https://status.flyio.net/incidents/3npj6935byt4
Most of the impacted clusters may see a degraded replica or PG Bouncer in their statuspage. A very small number may be unable to connect to their MPG primary node, the team is working to restore connectivity as the top priority. Users may also see delays registering new clusters in these regions at this time.