bcmwl-kernel-source fails to build on daily isos 20.04 & 22.04

[Otto Ersek]

hi :-)

the bcwml-kernel-source fails to build on the latest focal & jammy daily iso image

it seems that the apt repository on the daily images is being stale with respect to the livefs.

issue seems to be with:
dpkg-dev_1.21.1ubuntu2_all.deb

kind regards
Otto

[Otto Ersek (3rs3k)]

Hi :-)

today I tested both again: jammy,focal
http://cdimage.ubuntu.com/xubuntu/{jammy,focal}/daily-live/20220529/jammy-desktop-amd64.iso

on the daily focal even gcc fails to install
due to linux/linux-libc-dev_5.4.0-115.129_amd64.deb

kind regards

[Otto Ersek (3rs3k)]

kinetic fails as well,...

[Steve Langasek (vorlon)]

Similar to bug #1973884 but maybe this time we've caught it in the act.

-rw-rw-r-- 2 cdimage cdimage 267375 May 24 02:34 InRelease

$ grep -l 'Syncing .* mirror' */*/*20220524* | xargs ls -l | sort -k8
-rw-rw-r-- 1 cdimage cdimage 89708 May 24 02:04 xubuntu/kinetic/daily-live-20220524.log
-rw-rw-r-- 1 cdimage cdimage 89996 May 24 02:56 ubuntu-mate/kinetic/daily-live-20220524.log
-rw-rw-r-- 1 cdimage cdimage 357422 May 24 15:23 ubuntu-budgie/focal/daily-live-20220524.log
-rw-rw-r-- 1 cdimage cdimage 217210 May 24 16:51 lubuntu/focal/daily-live-20220524.log
-rw-rw-r-- 1 cdimage cdimage 235388 May 24 18:53 ubuntustudio/focal/dvd-20220524.log
-rw-rw-r-- 1 cdimage cdimage 411963 May 24 23:13 ubuntukylin/focal/daily-live-20220524.log
$

Ok, a lot of those are started later than the last updated InRelease file; what about the next day?

$ grep -l 'Syncing .* mirror' */*/*20220525* | xargs ls -l | sort -k8
-rw-rw-r-- 1 cdimage cdimage 89740 May 25 02:07 xubuntu/kinetic/daily-live-20220525.log
-rw-rw-r-- 1 cdimage cdimage 92561 May 25 03:21 ubuntu-mate/kinetic/daily-live-20220525.log
$

Again, the log for the next image build after the mirror sync shows nothing interesting.

ubuntu-mate-amd64 on Launchpad finished at 2022-05-25 03:13:42 (Successfully built)
===== Syncing Ubuntu-MATE mirror =====
Wed May 25 03:13:42 UTC 2022
===== Building britney =====
Wed May 25 03:18:51 UTC 2022

This time, there are no stuck invocations of for-project; the oldest of these processes now is 4 hours old.

But the latest build of an image that uses an apt pool still shows it stuck.

kubuntu-amd64 on Launchpad finished at 2022-05-29 05:50:35 (Successfully built)
===== Parallel build; waiting for Kubuntu mirror to sync =====
Sun May 29 05:50:35 UTC 2022
===== Building britney =====

The lockfile does not exist at this moment:

$ ls -l ~/cdimage/etc/.lock-archive-sync
ls: cannot access '/home/cdimage/cdimage/etc/.lock-archive-sync': No such file or directory
$

Will attempt another manual build to see what comes of it.

[Steve Langasek (vorlon)]

And in the log of this build, AFTER I confirmed that there was no lock file on disk:

ubuntu-arm64 on Launchpad finished at 2022-05-29 08:20:23 (Successfully built)
===== Parallel build; waiting for Ubuntu mirror to sync =====
Sun May 29 08:20:24 UTC 2022
===== Building britney =====

It seems something is wrong with the lockfile detection here.

[Steve Langasek (vorlon)]

or I'm just reading the code too fast.

    if not multipidfile_state:
[...]
    else:
        log_marker(
            "Parallel build; waiting for %s mirror to sync" % capproject)

Reading further now.

[Steve Langasek (vorlon)]

Right.

def build_image_set(config, options):
    """Master entry point for building images."""
    with handle_signals():
        multipidfile = MultiPIDFile(
            os.path.join(config.root, "etc", ".build-image-set-pids"))
        with lock_build_image_set(config):
            with multipidfile.held(os.getpid()) as multipidfile_state:
                return build_image_set_locked(
                    config, options, multipidfile_state)

So, .build-image-set-pids is not empty, and presumably has not been for some time (and which I would've recognized immediately from the last bug report if I had read the code right). Which means this actually is the same problem we've had before, where the server has gotten so busy with daily builds that it never quiesces and therefore never has a chance to sync the mirrors.

This problem is exacerbated by the fact that we now have riscv64 daily builds on multiple series, each of which takes many hours to complete and therefore holds the lock.

I would really like us to refactor our mirroring to be able to use by-hash indices for image builds and avoid the need for locking around the rsync, but that is a much bigger project. In the meantime we need to look instead for ways to reduce the contention. One option is to align the schedule for all the riscv64 builds to start at roughly the same time, so that the impact of these builds running for a long time is minimized.

[Otto Ersek (3rs3k)]

Dear Steve
Thx for your effort tracking this issue to its roots!

Kind regards
Otto

On Sun, 29 May 2022, 20:55 Steve Langasek, <email address hidden>
wrote:

> Right.
>
> def build_image_set(config, options):
> """Master entry point for building images."""
> with handle_signals():
> multipidfile = MultiPIDFile(
> os.path.join(config.root, "etc", ".build-image-set-pids"))
> with lock_build_image_set(config):
> with multipidfile.held(os.getpid()) as multipidfile_state:
> return build_image_set_locked(
> config, options, multipidfile_state)
>
> So, .build-image-set-pids is not empty, and presumably has not been for
> some time (and which I would've recognized immediately from the last bug
> report if I had read the code right). Which means this actually is the
> same problem we've had before, where the server has gotten so busy with
> daily builds that it never quiesces and therefore never has a chance to
> sync the mirrors.
>
> This problem is exacerbated by the fact that we now have riscv64 daily
> builds on multiple series, each of which takes many hours to complete
> and therefore holds the lock.
>
> I would really like us to refactor our mirroring to be able to use by-
> hash indices for image builds and avoid the need for locking around the
> rsync, but that is a much bigger project. In the meantime we need to
> look instead for ways to reduce the contention. One option is to align
> the schedule for all the riscv64 builds to start at roughly the same
> time, so that the impact of these builds running for a long time is
> minimized.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1976146
>
> Title:
> bcmwl-kernel-source fails to build on daily isos 20.04 & 22.04
>
> Status in Ubuntu CD Images:
> New
>
> Bug description:
> hi :-)
>
> the bcwml-kernel-source fails to build on the latest focal & jammy
> daily iso image
>
> it seems that the apt repository on the daily images is being stale
> with respect to the livefs.
>
> issue seems to be with:
> dpkg-dev_1.21.1ubuntu2_all.deb
>
> kind regards
> Otto
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-cdimage/+bug/1976146/+subscriptions
>
>

[Steve Langasek (vorlon)]

I've satisfied myself that removing the lock handling is the correct answer, because nowadays the rsync mirror source has a minimum retention policy of 2 days for packages to allow use with by-hash references, so an image build should never break due to a deleted file due to rsync running in parallel. Waiting for a +1 from others on the release team before I change the code for this.

[Otto Ersek (3rs3k)]

daily images for 20220530 are still out of sync,... NO stressing intended :-) !!!
I just wanted to drop the info,...

[Steve Langasek (vorlon)]

Code has landed to relax the lock handling. ubuntu/jammy/daily-live-20220531.log build log from a test build:

ubuntu-arm64 on Launchpad finished at 2022-05-31 06:52:45 (Successfully built)
===== Syncing Ubuntu mirror =====
Tue May 31 06:52:46 UTC 2022
===== Building britney =====
Tue May 31 06:57:56 UTC 2022

The build succeeded.

It must be noted that removing this locking DOES introduce a very small window when an rsync could break another in-progress image build, because although the byhash indices are kept around, debian-cd does not use them; so since the InRelease file is access first, followed by the Packages files, an rsync could race and cause the image build to access a too-new Packages file that fails checksum validation. The window when this could happen is fairly small and the side-effects of the current implementation are sufficiently buggy that I think we should still proceed with this code change.

Closing the bug.

[Otto Ersek (3rs3k)]

Hi!

I have just checked:

http://cdimage.ubuntu.com/xubuntu/daily-live/20220601/kinetic-desktop-amd64.iso
(still out of sync -> gcc fails, binutils)

http://cdimage.ubuntu.com/xubuntu/jammy/daily-live/20220601/jammy-desktop-amd64.iso
(still out of sync -> missing dpkg-dev_1.21.1ubuntu2.1_all.deb)

http://cdimage.ubuntu.com/xubuntu/focal/daily-live/20220601/focal-desktop-amd64.iso
ISO missing could not check

again NO stressing intended :-) !!!

kind regards
Otto

[Steve Langasek (vorlon)]

Ok, well, this is a different issue now! the rsync server is refusing connections from cdimage-master. I'll file an RT internally to get this resolved.

[Otto Ersek (3rs3k)]

Cool :-) thank you!
I'll check the images tomorrow again,...

On Wed, 1 Jun 2022, 09:50 Steve Langasek, <email address hidden>
wrote:

> Ok, well, this is a different issue now! the rsync server is refusing
> connections from cdimage-master. I'll file an RT internally to get this
> resolved.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1976146
>
> Title:
> bcmwl-kernel-source fails to build on daily isos 20.04 & 22.04
>
> Status in Ubuntu CD Images:
> Fix Released
>
> Bug description:
> hi :-)
>
> the bcwml-kernel-source fails to build on the latest focal & jammy
> daily iso image
>
> it seems that the apt repository on the daily images is being stale
> with respect to the livefs.
>
> issue seems to be with:
> dpkg-dev_1.21.1ubuntu2_all.deb
>
> kind regards
> Otto
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-cdimage/+bug/1976146/+subscriptions
>
>

[Steve Langasek (vorlon)]

This is RT #150983 internally.

[Otto Ersek (3rs3k)]

xubuntu:
kinetic daily: 20220602 is fine now :-) gcc + bcmwl builds fine
however focal and jammy dailies are MISSING which makes me a little bit nervous :-)

[Steve Langasek (vorlon)]

The RT ticket has been resolved and I can confirm rsync is working again. Images built from 02:08 UTC onwards should have up-to-date package pools.

[Otto Ersek (3rs3k)]

yes kinetic is fine

however the "last" iso for focal available is 2 days old
https://cdimage.ubuntu.com/xubuntu/focal/daily-live/20220531/focal-desktop-amd64.iso

however the "last" iso for focal available is 1 day old
https://cdimage.ubuntu.com/xubuntu/jammy/daily-live/20220601/jammy-desktop-amd64.iso

seems the daily build for 20220602 failed for both?
sorry for bothering you,...

[Otto Ersek (3rs3k)]

I mean there are NO builds for 20220602 for xubuntu focal and jammy

[Steve Langasek (vorlon)]

Yes, those builds failed because I accidentally introduced a regression where builds would fail if there was another rsync in progress. That's fixed now, so xubuntu will catch up with the next daily build.

[Otto Ersek (3rs3k)]

Dear Steve, thank you!

Kind regards
Otto

On Thu, 2 Jun 2022, 08:50 Steve Langasek, <email address hidden>
wrote:

> Yes, those builds failed because I accidentally introduced a regression
> where builds would fail if there was another rsync in progress. That's
> fixed now, so xubuntu will catch up with the next daily build.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1976146
>
> Title:
> bcmwl-kernel-source fails to build on daily isos 20.04 & 22.04
>
> Status in Ubuntu CD Images:
> Fix Released
>
> Bug description:
> hi :-)
>
> the bcwml-kernel-source fails to build on the latest focal & jammy
> daily iso image
>
> it seems that the apt repository on the daily images is being stale
> with respect to the livefs.
>
> issue seems to be with:
> dpkg-dev_1.21.1ubuntu2_all.deb
>
> kind regards
> Otto
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-cdimage/+bug/1976146/+subscriptions
>
>

[Otto Ersek (3rs3k)]

Hi
I have tested all 3 images now and everything is perfect!
thx