Delayed state updates: expect more proxy retries for instances of new deployments

Incident Report for Fly.io

Resolved

This incident has been resolved.

Posted Mar 05, 2023 - 21:11 CST

Update

State is a lot more consistent now and most problems should be resolved. Working on final consistency fixes.

Posted Mar 05, 2023 - 16:34 CST

Update

We are now restoring missing state information from every server, 1 by 1.

Posted Mar 05, 2023 - 10:12 CST

Update

We are aware some apps are still experiencing issues and are continuing work to fix state inconsistencies causing proxy timeouts.

Posted Mar 05, 2023 - 07:51 CST

Update

There are still some issues with older deployments, we are still cleaning up the state.

Posted Mar 04, 2023 - 22:45 CST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Mar 04, 2023 - 20:26 CST

Update

We are continuing to work on a fix for this issue.

Posted Mar 04, 2023 - 14:46 CST

Update

Apps v1 deploys have been disabled as we're working on restoring a working state.

Posted Mar 04, 2023 - 13:01 CST

Update

We're rolling back this update

Posted Mar 04, 2023 - 11:24 CST

Identified

An ongoing upgrade is causing delayed app instances state propagation

Posted Mar 04, 2023 - 08:46 CST

This incident affected: Platform and Tools (Deployments) and Edge Proxy (Routing).