Fly Metrics' ingestion is stopped working

Incident Report for Fly.io

Resolved

This incident has been resolved.

Posted Nov 05, 2024 - 23:47 UTC

Monitoring

We are processing the backlog and metrics are mostly backfilled.

Posted Nov 05, 2024 - 23:11 UTC

Update

Metrics writes have been queued and we are still processing them. We are going to scale up the cluster to catch up the backlog.

Posted Nov 05, 2024 - 21:25 UTC

Identified

The issue is in our Victoria Metrics cluster's storage nodes. We've upgraded and rebooted the nodes.

Posted Nov 05, 2024 - 19:30 UTC

Investigating

Fly Metrics's ingestion is stopped working. While you can access https://fly-metrics.net/, you may not able to see new data points since around 06:50 PM UTC. We are actively working on the issue.

Posted Nov 05, 2024 - 18:51 UTC

This incident affected: Metrics.