Delayed state causing more 502s and connection issues

Incident Report for Fly.io

Resolved

This incident has been resolved.

Posted Jul 01, 2023 - 14:10 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jul 01, 2023 - 02:53 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Jun 30, 2023 - 19:54 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jun 30, 2023 - 17:03 UTC

Identified

As we've grown, the stress on our state replication infrastructure became too great due to a mis-enforced concurrency limit. We're rolling out a fix to all hosts.

Posted Jun 30, 2023 - 15:47 UTC

Update

We are continuing to investigate this issue.

Posted Jun 30, 2023 - 15:07 UTC

Investigating

We are currently investigating this issue.

Posted Jun 30, 2023 - 13:44 UTC