Packet loss in London and Seattle

Incident Report for Fly.io

Resolved

This incident has been resolved.
Posted Mar 10, 2022 - 23:51 UTC

Update

This issue is largely resolved and has been for awhile, but we're not changing the status page until we've finished individually checking Postgres clusters.
Posted Mar 10, 2022 - 23:02 UTC

Update

We are continuing to monitor for any further issues.
Posted Mar 10, 2022 - 20:25 UTC

Update

We are continuing to monitor for any further issues.
Posted Mar 10, 2022 - 20:25 UTC

Update

We're checking Postgres clusters individually to make sure they're functioning as expected.
Posted Mar 10, 2022 - 18:49 UTC

Monitoring

Network connectivity has stabilized. We're investigating lingering cluster issues.
Posted Mar 10, 2022 - 18:43 UTC

Update

Ongoing packet loss issues in LHR will disrupt app deployments in this region, as worker machines attempt to pull containers from our registry and are impeded by network conditions. Work ongoing!
Posted Mar 10, 2022 - 18:19 UTC

Update

Network connectivity issues in LHR will impact Fly.io Postgres clusters, as Postgres instances will, because of packet loss, be impaired when talking to their cluster. We're working on resolving this issue presently.
Posted Mar 10, 2022 - 18:18 UTC

Identified

We've identified the issue and a fix is being implemented.
Posted Mar 10, 2022 - 18:10 UTC

Investigating

We've detected significant packet loss between various parts of the world and our LHR and SEA regions
Posted Mar 10, 2022 - 16:55 UTC
This incident affected: Regional Availability (LHR - London, United Kingdom, SEA - Seattle, Washington (US)).