Bluegreen healthchecks not passing

Incident Report for Fly.io

Resolved

This incident has been resolved.
Posted Jan 15, 2025 - 19:22 UTC

Monitoring

A fix has been implemented and bluegreen deploys are succeeding as expected. We're continuing to monitor deploys to ensure stability, but customers should see BlueGreen deploys succeed in all regions.
Posted Jan 15, 2025 - 18:44 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Jan 15, 2025 - 18:19 UTC

Update

We are seeing signs of recovery, with Bluegreen deployments succeeding for many customers. We are continuing to investigate the root cause of the issue.

Customers who still experience a Bluegreen deployment failure can retry using the rolling strategy with `fly deploy --strategy rolling`.
Posted Jan 15, 2025 - 18:11 UTC

Update

A temporary workaround for new deployments is to use rolling strategy: `fly deploy --strategy rolling`.
Posted Jan 15, 2025 - 14:41 UTC

Update

We are still investigating the issue.
Posted Jan 15, 2025 - 14:35 UTC

Investigating

When deploying with bluegreen strategy some green machines (new app version) won't pass healthchecks.
Temporary workaround: unless bluegreen is a must for your app you can temporarily deploy using a different strategy by `fly deploy --strategy NAME`.
Posted Jan 15, 2025 - 13:28 UTC
This incident affected: Deployments.