Increased error rates for registry.fly.io

Incident Report for Fly.io

Resolved

The registry is fully restored.
Posted Mar 16, 2023 - 23:37 UTC

Update

We are still not seeing any new issues reported and are planning to re-deploy the registry again to ensure all issues from earlier have been resolved.
Posted Mar 16, 2023 - 20:34 UTC

Monitoring

The registry has been recovered from a previous version and we've re-enabled deploys as we continue to monitor for new issues.
Posted Mar 16, 2023 - 18:54 UTC

Update

Users may also see an increase in error rates when attempting to create or update machines using the API.
Posted Mar 16, 2023 - 18:30 UTC

Identified

We've identified the issue and are working to rollback the most recent deploy of the registry.
Posted Mar 16, 2023 - 18:29 UTC

Investigating

We're seeing an increased number of errors or timeouts when pushing images to registry.fly.io during deploys.
Posted Mar 16, 2023 - 18:24 UTC