Load balanced IP routing issues

Incident Report for Fly.io

Resolved

At approximately 2:30UTC, we added newly acquired IP blocks to our anycast infrastructure. The new blocks surpassed a BGP announcement limit in place with an upstream provider. This triggered withdrawal of routes for some existing IP blocks.

We restored the missing routes at 2:35UTC. The effect on end users varied because BGP changes take time to propagate. In the worst cases, end users weren't able to connect to application IPs for 5 minutes.

Posted Jul 28, 2022 - 14:30 UTC