Delayed state updates: expect more proxy retries for instances of new deployments
Incident Report for Fly.io
Resolved
This incident has been resolved.
Posted Mar 05, 2023 - 21:11 CST
Update
State is a lot more consistent now and most problems should be resolved. Working on final consistency fixes.
Posted Mar 05, 2023 - 16:34 CST
Update
We are now restoring missing state information from every server, 1 by 1.
Posted Mar 05, 2023 - 10:12 CST
Update
We are aware some apps are still experiencing issues and are continuing work to fix state inconsistencies causing proxy timeouts.
Posted Mar 05, 2023 - 07:51 CST
Update
There are still some issues with older deployments, we are still cleaning up the state.
Posted Mar 04, 2023 - 22:45 CST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 04, 2023 - 20:26 CST
Update
We are continuing to work on a fix for this issue.
Posted Mar 04, 2023 - 14:46 CST
Update
Apps v1 deploys have been disabled as we're working on restoring a working state.
Posted Mar 04, 2023 - 13:01 CST
Update
We're rolling back this update
Posted Mar 04, 2023 - 11:24 CST
Identified
An ongoing upgrade is causing delayed app instances state propagation
Posted Mar 04, 2023 - 08:46 CST
This incident affected: Platform and Tools (Deployments).