Delayed Machine Registration + Token Errors

Incident Report for Fly.io

Resolved

This incident has been resolved.
Posted Feb 05, 2026 - 20:53 UTC

Monitoring

A fix has been deployed across all impacted hosts. We are seeing a sharp reduction in Token errors since 20:00 UTC and other metrics are recovering as well. We are continuing to monitor closely
Posted Feb 05, 2026 - 20:11 UTC

Update

We saw some improvement from the previous fix, however errors remained elevated on some hosts.
We have identified the root cause of the remaining errors as a communication issue between the hosts and our Token database. We are preparing a fix that should resolve these.
Posted Feb 05, 2026 - 19:03 UTC

Identified

We have rolled out an initial fix for the token issues and are monitoring for improvements.
Posted Feb 05, 2026 - 18:19 UTC

Investigating

While Machine registration error rates have improved, we are now seeing elevated error rates verifying user tokens during some actions.

Users may see errors like "failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens" when deploying or creating machines.

We are investigating
Posted Feb 05, 2026 - 17:42 UTC

Update

A fix has been rolled out and most hosts are registering machines as normal. A few hosts remain with elevated error rates, we are continuing to fix these.
Users who experience an error creating or deploying a new machine should re-try the operation.
Posted Feb 05, 2026 - 16:59 UTC

Identified

We have identified elevated error rates registering new machines with our global state tracking service on some hosts. We have identified the issue and are deploying a fix.

Users may have seen elevated machine create, start, or deployment failures over the past ~20 minutes.
Posted Feb 05, 2026 - 16:43 UTC
This incident affected: Machines API and Deployments.