Image pull failures in FRA/IAD/EWR

Incident Report for Fly.io

Resolved

This incident has been resolved.

Posted Apr 09, 2023 - 10:05 CDT

Monitoring

We've rolled back some infrastructure changes that may have allowed the registry to leak connections. Registry errors have returned to normal levels, and deploys are functioning properly in all regions.

Posted Apr 09, 2023 - 09:23 CDT

Identified

The registry is rejecting connections from some hosts in some regions. We're restarting processes, which seems to be helping reduce the error rate.

Posted Apr 09, 2023 - 07:06 CDT

Investigating

We are investigating an increased rate of image pull failures in a handful of regions. These are interfering with some customer deploys, and show as a failed VM in `fly status`.

Posted Apr 09, 2023 - 06:53 CDT

This incident affected: Platform and Tools (Deployments, Docker Registry).