Grafana log search not loading

Incident Report for Fly.io

Resolved

This incident has been resolved.
Posted Oct 07, 2025 - 00:04 UTC

Monitoring

New app logs are once again stably flowing into the fly-metrics.net dashboard. Users should see and be able to search all app logs from 22:30 UTC onwards. We are continuing to monitor the customer logs cluster to ensure continued stability.

Not all historical logs were able to be backfilled. Many users will see missing or incomplete logs for the period between 08:00 - 22:30 UTC Oct-06-2025.
Posted Oct 06, 2025 - 23:18 UTC

Update

We are continuing to backfill historic logs into fly-metrics.net. New logs will be accessible once the backfill is complete.
Posted Oct 06, 2025 - 19:50 UTC

Update

We have applied a fix and historic logs are now backfilling into fly-metrics.net. New logs will be accessible once the backfill is complete.

Customers can continue view live logs for their apps using the `flyctl logs` command and from the logs tab of their app page at Fly.io/dashboard.
Posted Oct 06, 2025 - 17:46 UTC

Update

We are continuing to work on a fix for this issue. Searchable logs in the fly-metrics.net Grafana board remain unavailable at this time.

Customers can view live logs for their apps using the `flyctl logs` command and from the logs tab of their app page at Fly.io/dashboard .
Posted Oct 06, 2025 - 15:55 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Oct 06, 2025 - 14:23 UTC

Investigating

Grafana is returning 5xx errors when loading logs.
Posted Oct 06, 2025 - 12:34 UTC
This incident affected: Logs.