Delayed state updates: expect more proxy retries for instances of new deployments

Incident Report for Fly.io

Resolved

This incident has been resolved.
Posted Mar 06, 2023 - 03:11 UTC

Update

State is a lot more consistent now and most problems should be resolved. Working on final consistency fixes.
Posted Mar 05, 2023 - 22:34 UTC

Update

We are now restoring missing state information from every server, 1 by 1.
Posted Mar 05, 2023 - 16:12 UTC

Update

We are aware some apps are still experiencing issues and are continuing work to fix state inconsistencies causing proxy timeouts.
Posted Mar 05, 2023 - 13:51 UTC

Update

There are still some issues with older deployments, we are still cleaning up the state.
Posted Mar 05, 2023 - 04:45 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 05, 2023 - 02:26 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 04, 2023 - 20:46 UTC

Update

Apps v1 deploys have been disabled as we're working on restoring a working state.
Posted Mar 04, 2023 - 19:01 UTC

Update

We're rolling back this update
Posted Mar 04, 2023 - 17:24 UTC

Identified

An ongoing upgrade is causing delayed app instances state propagation
Posted Mar 04, 2023 - 14:46 UTC
This incident affected: Deployments.