Host failure in ORD

Incident Report for Fly.io

Resolved

All remaining machines have been migrated.

Posted Dec 08, 2022 - 15:33 CST

Update

All Postgres apps and pg_data volumes recreated from latest available snapshot were moved last night. Only a few remaining machines are left to migrate.

Posted Dec 07, 2022 - 10:17 CST

Monitoring

Applications without volumes have all been restarted on other hosts in the region. Most volumes have been restored from their most recent daily snapshot and are migrated to another host. Recently-written data will be unavailable for volumes not in a multi-node cluster.

Posted Dec 05, 2022 - 21:32 CST

Identified

We are working to replace a failed disk.

Posted Dec 05, 2022 - 15:42 CST

Investigating

We are troubleshooting a failed host in ORD.

Posted Dec 05, 2022 - 15:24 CST