Resolved
This incident has been resolved.
Posted Oct 13, 2021 - 17:42 UTC
Update
Deployments are working, but the API doesn't return any instances. This means your deployment will get stuck in the "Monitoring Deployment" phase. This does not mean it hasn't already succeeded.
Posted Oct 13, 2021 - 14:34 UTC
Monitoring
API and related services restored.
Posted Oct 13, 2021 - 10:48 UTC
Update
We are continuing work to restore our API.
Posted Oct 13, 2021 - 09:49 UTC
Update
The Fly secrets service is restored and applications are recovering. We're working to restore our API now.
Posted Oct 13, 2021 - 09:35 UTC
Update
Fly secrets are still recovering from the network disruption, apps that use secrets (including Fly.io itself) may not boot properly until the secrets service is fully restored.
Posted Oct 13, 2021 - 09:06 UTC
Update
We are currently restoring application connectivity. Some regions (including Sydney) are still experiencing service discovery and orchestration disruptions.
Posted Oct 13, 2021 - 08:40 UTC
Identified
Network connectivity restored to orchestration and certificate storage. App error rates are declining and we are bringing orchestration back online now.
Posted Oct 13, 2021 - 08:26 UTC
Investigating
We are experiencing a large scale network disruption affecting with our secrets storage and orchestration infrastructure. These are causing connection errors from some parts of the world, deployment failures, and other intermittent issues.
Posted Oct 13, 2021 - 07:47 UTC
This incident affected: Regional Availability (AMS - Amsterdam, Netherlands, ATL - Atlanta, Georgia (US), DFW - Dallas, Texas (US), EZE - Ezeiza, Argentina, FRA - Frankfurt, Germany, HKG - Hong Kong, IAD - Ashburn, Virginia (US), LAX - Los Angeles, California (US), NRT - Tokyo, Japan, ORD - Chicago, Illinois (US), SEA - Seattle, Washington (US), SIN - Singapore, SJC - San Jose, California (US), SYD - Sydney, Australia, YYZ - Toronto, Canada), DNS (Fly Machine .internal DNS), and Dashboard, Machines API, Persistent Storage (Volumes), Deployments, Logs.