This incident has been resolved.

A fix has been deployed across all impacted hosts. We are seeing a sharp reduction in Token errors since 20:00 UTC and other metrics are recovering as well. We are continuing to monitor closely

We saw some improvement from the previous fix, however errors remained elevated on some hosts. We have identified the root cause of the remaining errors as a communication issue between the hosts and our Token database. We are preparing a fix that should resolve these.

We have rolled out an initial fix for the token issues and are monitoring for improvements.

While Machine registration error rates have improved, we are now seeing elevated error rates verifying user tokens during some actions.

Users may see errors like "failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens" when deploying or creating machines.

We are investigating

A fix has been rolled out and most hosts are registering machines as normal. A few hosts remain with elevated error rates, we are continuing to fix these. Users who experience an error creating or deploying a new machine should re-try the operation.