High error rates across multiple regions

Incident Report for Fly.io

Resolved

This incident has been resolved.
Posted May 27, 2020 - 20:19 UTC

Update

All services restored. We are monitoring to make sure customer apps recovered gracefully.
Posted May 27, 2020 - 20:18 UTC

Monitoring

We've restored backhaul between all regions, error rates are decreasing. We are restarting numerous services to flush out old backhaul configurations.
Posted May 27, 2020 - 20:16 UTC

Update

We are continuing to investigate this issue.
Posted May 27, 2020 - 20:13 UTC

Update

Backhaul between regions is unstable and causing multiple connection failures.
Posted May 27, 2020 - 20:11 UTC

Investigating

Our monitoring is reporting errors across multiple Fly regions, we're investigating.
Posted May 27, 2020 - 20:05 UTC
This incident affected: Regional Availability (AMS - Amsterdam, Netherlands, ATL - Atlanta, Georgia (US), DFW - Dallas, Texas (US), EZE - Ezeiza, Argentina, FRA - Frankfurt, Germany, HKG - Hong Kong, IAD - Ashburn, Virginia (US), LAX - Los Angeles, California (US), NRT - Tokyo, Japan, ORD - Chicago, Illinois (US), SEA - Seattle, Washington (US), SIN - Singapore, SJC - San Jose, California (US), SYD - Sydney, Australia, YYZ - Toronto, Canada).