Resolved
This incident has been resolved.
Monitoring
With the additional capacity we've brought online, machine start failure rates in IAD have now recovered. We'll continue to monitor IAD capacity.
Identified
We've brought some additional capacity online in IAD and are seeing improvements, and we're continuing to work on adding more and freeing up additional room.
Investigating
We're continuing to evaluate our options for increasing short-term capacity in the IAD region.
Investigating
We're currently investigating capacity issues in IAD that is preventing machine starts (machine creates are currently unaffected). This may result in deploys failing to complete (even for apps outside of the IAD region). As a workaround, using legacy Fly builders explicitly located in another region (i.e., FLY_REMOTE_BUILDER_REGION=lhr fly deploy --depot=false --recreate-builder) may help in the meantime.