SSL, logs, orchestration errors
This incident has been resolved.
Deployments are working, but the API doesn't return any instances. This means your deployment will get stuck in the "Monitoring Deployment" phase. This does not mean it hasn't already succeeded.
API and related services restored.
We are continuing work to restore our API.
The Fly secrets service is restored and applications are recovering. We're working to restore our API now.
Fly secrets are still recovering from the network disruption, apps that use secrets (including Fly.io itself) may not boot properly until the secrets service is fully restored.
We are currently restoring application connectivity. Some regions (including Sydney) are still experiencing service discovery and orchestration disruptions.
Network connectivity restored to orchestration and certificate storage. App error rates are declining and we are bringing orchestration back online now.
We are experiencing a large scale network disruption affecting with our secrets storage and orchestration infrastructure. These are causing connection errors from some parts of the world, deployment failures, and other intermittent issues.
This incident affected: Application Hosting (EWR1 Application Hosting (New Jersey), AMS1 Application Hosting (Amsterdam), YYZ1 Application Hosting (Toronto), SJC1 Application Hosting (San Jose, CA), NRT1 Application Hosting (Tokyo, Japan), SEA1 Application Hosting (Seattle, Washington), LAX1 Application Hosting (Los Angeles, CA), IAD1 Application Hosting (Ashburn, Virginia), ORD1 Application Hosting (Chicago, Illinois), ATL1 Application Hosting (Atlanta, Georgia), DFW2 Application Hosting (Dallas, Texas), FRA1 Application Hosting (Frankfurt, Germany), HKG1 Application Hosting (Hong Kong), SYD1 Application Hosting (Sydney, Australia), MRS1 Application Hosting (Marseille, France), SIN1 Application Hosting (Singapore), Persistent Storage (Volumes), Private DNS) and Platform and Tools (API, Deployments, Dashboard, Logs).