Fly Registry Uploads Failing
Incident Report for Fly.io
Resolved
The incident has been resolved and we're not seeing any further issues with the registry.

Around 14:00 UTC a litefs cluster that supports the registry began lagging due to too many open files on individual hosts. We resolved the open file issue and recovered the litefs cluster, which was completed around 14:45 UTC.

Between 14:00 and 14:45 UTC deployments may have failed to push images to the registry. This prevented machines and nomad allocs from getting updates, and did not interfere with existing machine & nomad alloc operation.
Posted Sep 20, 2023 - 15:47 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 20, 2023 - 14:46 UTC
Investigating
We're investigating issues with the registry failing to allow new images to be uploaded.
Posted Sep 20, 2023 - 12:00 UTC