Deployment issues

Incident Report for Fly.io

Resolved

This incident has been resolved.

Posted Jun 02, 2022 - 18:57 UTC

Update

We are continuing to monitor for any further issues.

Posted Jun 02, 2022 - 18:12 UTC

Monitoring

We restored the Consul cluster and are monitoring individual services. Most service is restored, but some lingering issues may persist.

Posted Jun 02, 2022 - 17:30 UTC

Identified

Our Consul servers are in out-of-memory loops. We are working to recover them. Consul issues can cause problems with internal DNS resolution, deploys, and API access. We are working to recover the servers.

Posted Jun 02, 2022 - 17:19 UTC

Investigating

Our internal server-orchestration cluster is suffering from capacity issues, causing deployment operations to be unavailable.

Posted Jun 02, 2022 - 17:08 UTC

This incident affected: DNS (Fly Machine .internal DNS) and Dashboard, Machines API, Deployments.