SSL, logs, orchestration errors
Resolved
This incident has been resolved.
Posted Oct 13, 2021 - 12:42 CDT
Update
Deployments are working, but the API doesn't return any instances. This means your deployment will get stuck in the "Monitoring Deployment" phase. This does not mean it hasn't already succeeded.
Posted Oct 13, 2021 - 09:34 CDT
Monitoring
API and related services restored.
Posted Oct 13, 2021 - 05:48 CDT
Update
We are continuing work to restore our API.
Posted Oct 13, 2021 - 04:49 CDT
Update
The Fly secrets service is restored and applications are recovering. We're working to restore our API now.
Posted Oct 13, 2021 - 04:35 CDT
Update
Fly secrets are still recovering from the network disruption, apps that use secrets (including Fly.io itself) may not boot properly until the secrets service is fully restored.
Posted Oct 13, 2021 - 04:06 CDT
Update
We are currently restoring application connectivity. Some regions (including Sydney) are still experiencing service discovery and orchestration disruptions.
Posted Oct 13, 2021 - 03:40 CDT
Identified
Network connectivity restored to orchestration and certificate storage. App error rates are declining and we are bringing orchestration back online now.
Posted Oct 13, 2021 - 03:26 CDT
Investigating
We are experiencing a large scale network disruption affecting with our secrets storage and orchestration infrastructure. These are causing connection errors from some parts of the world, deployment failures, and other intermittent issues.
Posted Oct 13, 2021 - 02:47 CDT
This incident affected: Application Hosting (EWR1 Application Hosting (New Jersey), AMS1 Application Hosting (Amsterdam), YYZ1 Application Hosting (Toronto), SJC1 Application Hosting (San Jose, CA), NRT1 Application Hosting (Tokyo, Japan), SEA1 Application Hosting (Seattle, Washington), LAX1 Application Hosting (Los Angeles, CA), IAD1 Application Hosting (Ashburn, Virginia), ORD1 Application Hosting (Chicago, Illinois), ATL1 Application Hosting (Atlanta, Georgia), DFW2 Application Hosting (Dallas, Texas), FRA1 Application Hosting (Frankfurt, Germany), HKG1 Application Hosting (Hong Kong), SYD1 Application Hosting (Sydney, Australia), MRS1 Application Hosting (Marseille, France), SIN1 Application Hosting (Singapore), Persistent Storage (Volumes), Private DNS) and Platform and Tools (API, Deployments, Dashboard, Logs).