Ubuntu image builds broken for powerpc on xenial

[Robert C Jennings]

[Impact]

Currenty powerpc image builds are failing on xenial. The main reason is that snapd now provides a powerpc 'empty' dummy package (as the architecture is not supported) but still ships regular maintainer scripts that try to do operations on installed files (which are not present due to the empty package). This basically makes the dummy snapd powerpc package uninstallable. And since launchpad-buildd pulls in snapd by default, this breaks any powerpc image builds currently.

Also, besides that, it seems that livecd-rootfs was directly depending on snapd as well. To save ourselves trouble, let's make that dependency arch-specific.

[Test Case]

Build test images on powerpc, those should now succeed.

A quick sanity run of snapd on non-powerpc architectures should also be performed, making sure that the no-change rebuild for those did not regress.

[Regression Potential]

None. This is only an arch-specific dependency change in livecd-rootfs and maintainer-script changes in snapd. It is highly unlikely that any of this could have effect in anything other than powerpc (which is not supported anyway).

The no-change rebuild of snapd might, as any rebuild, potentially regress, but the risk is very very low as the previous snapd has been built and updated recently (so not too many archive changes).

[Original Description]

livecd-rootfs is used to build our Ubuntu images and it depends on ubuntu-image. With ubuntu-image 1.7 being published to xenial-updates the diff
shows that the versioned dependency on snapd was bumped to >= 2.38; which is not satisfiable on powerpc because snapd is not buildable on powerpc with gccgo (vs golang).

This affects xenial cloud image builds.

Original description:

This commit https://git.launchpad.net/livecd-rootfs/commit/debian/control?h=ubuntu/xenial&id=b8b2e343827adb9020d5b71f232c0361bd72685f added a depends on ubuntu-image but that package is not available on xenial for powerpc. So powerpc image builds are failing. :(

[Robert C Jennings (rcj)]

RUN: /usr/share/launchpad-buildd/bin/in-target buildlivefs --backend=lxd --series=xenial --arch=powerpc LIVEFSBUILD-167495 --project ubuntu-cpc --datestamp 20190523 --image-format ext4
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 livecd-rootfs : Depends: ubuntu-image but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Install failed

[Steve Langasek (vorlon)]

That commit has been in xenial since October and is not the proximate cause of livecd-rootfs being uninstallable on powerpc. The timeline (breaking on the 23rd) DOES align with ubuntu-image 1.7 being published to xenial-updates. The diff of that ubuntu-image shows that the versioned dependency on snapd is bumped to be >= 2.38; which is not satisfiable on powerpc because snapd is not buildable on powerpc with gccgo (vs golang).

I believe this will be fixed in the next upload of snapd, which will make snapd an empty package on powerpc instead of failing to build.

[Michael Vogt (mvo)]

We believe this will be fixed with https://github.com/snapcore/snapd/pull/6928 - and we can cherry pick into the 2.39.1 SRU

[Łukasz Zemczak (sil2100)]

In the meantime, as per Steve's valid recommendation, I'll make the snapd build dependency for 1.7 in xenial optional for powerpc.

[Steve Langasek (vorlon)]

Since snaps are not supported on powerpc, it also doesn't make sense to have an ubuntu-image package that depends on snapd on this architecture. It is still potentially interesting to have ubuntu-image available on all archs for the 'ubuntu-image classic' use case, so in terms of overall direction I think it's better to have ubuntu-image drop its dep on snapd on powerpc and NOT have livecd-rootfs drop its dep on ubuntu-image on powerpc, in addition to having snapd available again on powerpc as a dummy package.

It is very unlikely that we will ever SRU back to xenial any support for actual ubuntu-image classic images on powerpc, but just in case this would future-proof it.

[Steve Langasek (vorlon)]

marking invalid for the devel series because powerpc has been dropped as an arch post-xenial.

[Łukasz Zemczak (sil2100)]

...actually we can't really switch the ubuntu-image dependency (it's an arch: all package, duh). But we can get rid of the ubuntu-image dep in livecd-rootfs on powerpc.

[Steve Langasek (vorlon)]

Hello Robert, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.48 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

autopkgtests pass, the only delta is to drop a dependency on one architecture, considering verified.

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.408.48

---------------
livecd-rootfs (2.408.48) xenial; urgency=medium

  * Do not depend on ubuntu-image on powerpc as it is not a supported platform
    for building ubuntu-core images (and we don't build classic preinstalled
    images on xenial). (LP: #1830823)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 29 May 2019 18:50:31 +0200

[Steve Langasek (vorlon)]

livecd-rootfs still doesn't work on powerpc, because now that we are not depending on ubuntu-image, python3-yaml is not pulled in as an indirect dependency, leading to runtime failures.

Will reupload a further fix.

[Brian Murray (brian-murray)]

Hello Robert, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.49 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

livecd-rootfs autopkgtests again passing.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.408.49

---------------
livecd-rootfs (2.408.49) xenial; urgency=medium

  * Add explicit dependency on python3-yaml, which is used in the code but
    was previously pulled in indirectly via ubuntu-image. LP: #1830823.

 -- Steve Langasek <email address hidden> Thu, 30 May 2019 14:20:30 -0700

[Michael Vogt (mvo)]

We fixed the build of snapd on powerpc now - we build an mostly empty package there. Its in https://launchpad.net/ubuntu/xenial/+queue?queue_state=1&queue_text=snapd right now.

[David Krauser (davidkrauser)]

We're still seeing powerpc builds failing on xenial: https://i430477153.restricted.launchpadlibrarian.net/430477153/buildlog_ubuntu_xenial_powerpc_powerpc_
cpc_BUILDING.txt.gz?token=cXQZLr4n6nn9LNfgRxL1Q5LX0zCzhS2z

[David Krauser (davidkrauser)]

line wrap killed the link: https://i430477153.restricted.launchpadlibrarian.net/430477153/buildlog_ubuntu_xenial_powerpc_powerpc_cpc_BUILDING.txt.gz?token=cXQZLr4n6nn9LNfgRxL1Q5LX0zCzhS2z

[David Krauser (davidkrauser)]

The specific error:

Setting up snapd (2.39.2) ...
md5sum: /etc/apparmor.d/usr.lib.snapd.snap-confine: No such file or directory
chmod: cannot access '/var/lib/snapd/void': No such file or directory
dpkg: error processing package snapd (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of livecd-rootfs:
 livecd-rootfs depends on snapd; however:
  Package snapd is not configured yet.

dpkg: error processing package livecd-rootfs (--configure):

[Brian Murray (brian-murray)]

Hello Robert, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.50 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

I have just been told by Rob that powerpc builds are still failing with the new livecd-rootfs as apparently some other package still pulls in snapd. I guess we might have to fix this in snapd then.

@mvo Could you take a look and see what's up with the new dummy snapd package for powerpc?

[Robert C Jennings (rcj)]

If I understand launchpad-buildd[1] correctly, snapd will be installed regardless of livecd-rootfs' dependencies. The fix needs to be in snapd for powerpc on xenial. I confirmed that with a -proposed build with livecd-rootfs 2.408.50 we will still try to install snapd and fail the build as before.

[1] https://bazaar.launchpad.net/~canonical-launchpad-branches/launchpad-buildd/trunk/view/head:/lpbuildd/target/build_livefs.py#L89

[Robert C Jennings (rcj)]

I've marked this as verification failed as the build is not fixed by this change. The livecd-rootfs change looks safe enough but does not address root cause and as such we can not complete a build to verify this change doesn't break things further; all we can say is that it doesn't address root cause.

[Łukasz Zemczak (sil2100)]

I have uploaded the cherry-pick of snapd (https://github.com/snapcore/snapd/pull/7038) to xenial. I would still like to get the livecd-rootfs change released along with snapd as the snapd dependency *should* be arch-specific anyway. There is no user-impact in getting livecd-rootfs released along with snapd (it's a package only used during builds).

[Steve Langasek (vorlon)]

Hello Robert, or anyone else affected,

Accepted snapd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapd/2.39.2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andy Whitcroft (apw)]

Hello Robert, or anyone else affected,

Accepted snapd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapd/2.39.2ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Ok, finally snapd built successfully for all arches.

@rcj could you try building xenial powerpc images and seeing if snapd still fails to install properly?

I'll ask cachio or someone else from the snapd team to do a quick sanity test of the snapd rebuilt binaries on other arches to make sure there's no regressions introduced through dependencies.

[Robert C Jennings (rcj)]

I have built using --proposed for powerpc and amd64 using the ubuntu-cpc project. I'm marking this as verification done.

[Łukasz Zemczak (sil2100)]

Ok, snapd team also ran some sanity tests on the new xenial-proposed snapd and did not encounter any issues. Let me release this early.

[Launchpad Janitor (janitor)]

This bug was fixed in the package snapd - 2.39.2ubuntu0.2

---------------
snapd (2.39.2ubuntu0.2) xenial; urgency=medium

  * Cherry-pick pull-request 7040 fixing build-time unit-tests that started
    failing recently.

snapd (2.39.2ubuntu0.1) xenial; urgency=medium

  * Cherry-pick pull-request 7038 to make sure maintainer scripts do nothing
    on powerpc (we only ship an empty package there). (LP: #1830823)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Thu, 27 Jun 2019 11:58:39 +0200

[Brian Murray (brian-murray)]

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.408.50

---------------
livecd-rootfs (2.408.50) xenial; urgency=medium

  * Actually, do not depend on snapd on powerpc as well. Snaps are not
    supported on this platform at all. (LP: #1830823)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 25 Jun 2019 22:54:10 +0200