Fly Metrics' ingestion is stopped working
Incident Report for Fly.io
Resolved
This incident has been resolved.
Posted Nov 05, 2024 - 23:47 UTC
Monitoring
We are processing the backlog and metrics are mostly backfilled.
Posted Nov 05, 2024 - 23:11 UTC
Update
Metrics writes have been queued and we are still processing them. We are going to scale up the cluster to catch up the backlog.
Posted Nov 05, 2024 - 21:25 UTC
Identified
The issue is in our Victoria Metrics cluster's storage nodes. We've upgraded and rebooted the nodes.
Posted Nov 05, 2024 - 19:30 UTC
Investigating
Fly Metrics's ingestion is stopped working. While you can access https://fly-metrics.net/, you may not able to see new data points since around 06:50 PM UTC. We are actively working on the issue.
Posted Nov 05, 2024 - 18:51 UTC
This incident affected: Metrics.