Consul cluster outage
Incident Report for Fly.io
Resolved
This incident has been resolved.
Posted Mar 15, 2023 - 22:28 CDT
Monitoring
Our Consul cluster is now stable again. We've re-enabled Nomad deployments and are monitoring for any further issues.
Posted Mar 15, 2023 - 19:58 CDT
Update
The Consul cluster rebuild is nearly completed and we are working to restore service.
Posted Mar 15, 2023 - 19:04 CDT
Update
The Consul cluster rebuild is nearly completed and we are working to restore service.
Posted Mar 15, 2023 - 18:53 CDT
Update
We are working to build a new Consul cluster with 10x the RAM. We aren't yet sure, but believe a routine DNS change might have created a thundering herd problem causing Consul servers to immediately increase RAM usage by 500%. This is not ideal.
Posted Mar 15, 2023 - 17:42 CDT
Identified
We're still working on recovering our Consul cluster.
Posted Mar 15, 2023 - 16:10 CDT
Monitoring
Our Consul cluster is experiencing an outage. This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps.
Posted Mar 15, 2023 - 14:20 CDT
This incident affected: Platform and Tools (API, Deployments) and Edge Proxy (Routing).