SSL, logs, orchestration errors
Resolved
This incident has been resolved.
Posted Oct 13, 2021 - 12:42 CDT
Update
Deployments are working, but the API doesn't return any instances. This means your deployment will get stuck in the "Monitoring Deployment" phase. This does not mean it hasn't already succeeded.
Posted Oct 13, 2021 - 09:34 CDT
Monitoring
API and related services restored.
Posted Oct 13, 2021 - 05:48 CDT
Update
We are continuing work to restore our API.
Posted Oct 13, 2021 - 04:49 CDT
Update
The Fly secrets service is restored and applications are recovering. We're working to restore our API now.
Posted Oct 13, 2021 - 04:35 CDT
Update
Fly secrets are still recovering from the network disruption, apps that use secrets (including Fly.io itself) may not boot properly until the secrets service is fully restored.
Posted Oct 13, 2021 - 04:06 CDT
Update
We are currently restoring application connectivity. Some regions (including Sydney) are still experiencing service discovery and orchestration disruptions.
Posted Oct 13, 2021 - 03:40 CDT
Identified
Network connectivity restored to orchestration and certificate storage. App error rates are declining and we are bringing orchestration back online now.
Posted Oct 13, 2021 - 03:26 CDT
Investigating
We are experiencing a large scale network disruption affecting with our secrets storage and orchestration infrastructure. These are causing connection errors from some parts of the world, deployment failures, and other intermittent issues.
Posted Oct 13, 2021 - 02:47 CDT
This incident affected: Application Hosting (AMS - Amsterdam, Netherlands, DFW - Dallas, Texas (US), EWR - Secaucus, New Jersey (US), FRA - Frankfurt, Germany, HKG - Hong Kong, IAD - Ashburn, Virginia (US), LAX - Los Angeles, California (US), NRT - Tokyo, Japan, ORD - Chicago, Illinois (US), SEA - Seattle, Washington (US), SIN - Singapore, SJC - Sunnyvale, California (US), SYD - Sydney, Australia, YYZ - Toronto, Canada, ATL - Atlanta, Georgia (US), MRS - Marseille, France, Persistent Storage (Volumes), Private DNS) and Platform and Tools (API, Deployments, Dashboard, Logs).