Comment 6 for bug 1976146

Right.

def build_image_set(config, options):
    """Master entry point for building images."""
    with handle_signals():
        multipidfile = MultiPIDFile(
            os.path.join(config.root, "etc", ".build-image-set-pids"))
        with lock_build_image_set(config):
            with multipidfile.held(os.getpid()) as multipidfile_state:
                return build_image_set_locked(
                    config, options, multipidfile_state)

So, .build-image-set-pids is not empty, and presumably has not been for some time (and which I would've recognized immediately from the last bug report if I had read the code right). Which means this actually is the same problem we've had before, where the server has gotten so busy with daily builds that it never quiesces and therefore never has a chance to sync the mirrors.

This problem is exacerbated by the fact that we now have riscv64 daily builds on multiple series, each of which takes many hours to complete and therefore holds the lock.

I would really like us to refactor our mirroring to be able to use by-hash indices for image builds and avoid the need for locking around the rsync, but that is a much bigger project. In the meantime we need to look instead for ways to reduce the contention. One option is to align the schedule for all the riscv64 builds to start at roughly the same time, so that the impact of these builds running for a long time is minimized.