Resolved
This incident has been resolved.
Monitoring
Our Consul cluster is now stable again. We've re-enabled Nomad deployments and are monitoring for any further issues.
Identified
The Consul cluster rebuild is nearly completed and we are working to restore service.
Identified
The Consul cluster rebuild is nearly completed and we are working to restore service.
Identified
We are working to build a new Consul cluster with 10x the RAM. We aren't yet sure, but believe a routine DNS change might have created a thundering herd problem causing Consul servers to immediately increase RAM usage by 500%. This is not ideal.
Identified
We're still working on recovering our Consul cluster.
Monitoring
Our Consul cluster is experiencing an outage. This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps.