Resolved
This incident has been resolved. We have seen API latency normalize and remain normal since ~19:00 UTC.
Monitoring
API error rates have normalized, however users may still see elevated latency reaching some GraphQL endpoints. Latency continues to trend in the right direction, we continue to monitor for full recovery .
Identified
We have deployed an initial fix and are seeing improvements. GraphQL error rates and latency remain elevated over the baseline at this time. We are continuing to keep a close eye on recovery.
Identified
The issue has been identified and a fix is being implemented.
Investigating
We are continuing to investigate elevated latency and error rates on our GraphQL API endpoints. Users may see errors on parts of the platform that use these APIs. This includes Flyctl actions such as deploys, as well as the fly.io dashboard.
Investigating
We are investigating elevated API Latency and Error rates on the platform. Users may see delays or errors creating apps, as well as on some dashboard pages.
Machines API actions appear unimpacted at this time