We updated our remote builders to load their images from docker-hub-mirror.fly.io/flyio/rchab, which is a caching mirror of registry-1.docker.io. We're seeing remote builders more reliably work.
Deployments that use an image from the Docker Hub Registry—any unqualified FROM in a Dockerfile or setting an unqualified image name with `fly deploy --image`—may still fail.
You can try adding docker-hub-mirror.fly.io/IMAGE_NAME to your Dockerfile or the `--image` flag on `fly deploy`. That will work more reliably if we have a cached copy of the image from Docker Hub.
The `fly deploy --local-only` work around is still a good option, if it works for you.
Another work around is finding another registry, like ghcr.io or quay.io or something like that, which has a copy of the image you would normally get from the Docker Hub Registry.
Marking this as monitoring while we wait for further updates from Docker Hub via their status page.
Deployments that use an image from the Docker Hub Registry—any unqualified FROM in a Dockerfile or setting an unqualified image name with `fly deploy --image`—may fail.
Existing machines and allocs are not affected. Creating apps, machines, or allocs from images in the Fly Registry (registry.fly.io/APP_NAME) are working.
Remote builders may fail to download the image from Docker Hub. We're working on a fix for that now.
Try `fly deploy --local-only` if you experience remote builder issues as result of this issue. That will use a local docker environment on your system to build and push the image. That might work and work around the remote builder issue for now, though we understand local docker environments can be unreliable in certain cases (for example, building linux/amd64 images on macOS M1/M2 chips can seqfault with a known wontfix issue in qemu).
Posted Sep 29, 2023 - 02:45 UTC
This incident affected: Platform and Tools (Deployments).