Support generating a rootfs tree only

[Łukasz Zemczak]

[Impact]

For the use of ubuntu-image classic image generation, it is required that livecd-rootfs supports a way to only generate the rootfs tree for a selected set of arguments (project, architecture etc.). By design ubuntu-image will use livecd-rootfs for preparation of the rootfs contents and then continue with building the image. Since ubuntu-image is backported feature-complete into artful, zesty and xenial, we would need the same change to be propagated to those stable series.

[Test Case]

 * Install livecd-rootfs from -proposed
 * Create a temporary directory, cd into it
 * cp -a `dpkg -L livecd-rootfs | grep "auto$"` .
 * sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=<SERIES> lb config
 * sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=<SERIES> lb build
 * Make sure the build succeeds and that the current directory will have a chroot/ with the build rootfs contents
 * Make sure that there config/hooks/ directory has no binary hooks

[Regression Potential]

The proposed change should be self-contained and not affecting any other code paths, but in the case of a very unlikely situation regular image builds could be broken by binary hooks not being executed.

[Łukasz Zemczak (sil2100)]

livecd-rootfs (2.480) bionic; urgency=medium

  [ Gary Wang ]
  * Add the IMAGEFORMAT=none to support generating a single rootfs.

 -- Steve Langasek <email address hidden> Mon, 06 Nov 2017 11:12:45 -0800

[Steve Langasek (vorlon)]

To my understanding, the other piece of this in livecd-rootfs - support for setting up the trees and invoking ubuntu-image as part of a livefs build - has not yet landed. Should it, before we settle on SRUing this?

[Łukasz Zemczak (sil2100)]

In my opinion it is not required for us to wait on that as this change is not needed for ubuntu-image classic build support to work. Without the support for invoking ubuntu-image for builds in livecd-rootfs users can still invoke ubuntu-image directly to get a working classic image generated. This way we can release ubuntu-image, get more global testing from users and only then switch certain projects in livecd-rootfs to use ubuntu-image instead of the regular mechanisms.

[Steve Langasek (vorlon)]

On Wed, Nov 08, 2017 at 10:09:42AM -0000, Łukasz Zemczak wrote:
> In my opinion it is not required for us to wait on that as this change
> is not needed for ubuntu-image classic build support to work. Without
> the support for invoking ubuntu-image for builds in livecd-rootfs users
> can still invoke ubuntu-image directly to get a working classic image
> generated. This way we can release ubuntu-image, get more global testing
> from users and only then switch certain projects in livecd-rootfs to use
> ubuntu-image instead of the regular mechanisms.

My point is that "switching projects in livecd-rootfs" involves additional
development work that has not yet been done; you can't just invoke
ubuntu-image for a classic image build yet because you need to set up the
gadget tree, which there is no code yet to facilitate.

[Łukasz Zemczak (sil2100)]

You can invoke ubuntu-image for a classic image build right now. Yes, you need a specific gadget tree, but it doesn't require any additional special infrastructure to get such a gadget tree prepared. For instance, for testing purposes we already use a custom gadget tree that's available https://github.com/sil2100/pc-amd64-gadget . I would not like to block 1.2 on something that is not required for ubuntu-image to work. Yes, it would be more convenient to have livecd-rootfs do the gadget tree setup for people, but ubuntu-image is designed such that one can just run it by hand with a selected gadget tree and get the resulting image.

For instance, with the current livecd-rootfs you get a working classic image if you run this:
sudo ./ubuntu-image classic --project ubuntu-cpc https://github.com/sil2100/pc-amd64-gadget.git

I see it this way: for snap-based images we also don't require people to run ubuntu-image through livecd-rootfs, manually you also have to somehow 'fetch' by yourself a fitting model assertion and feed it to the command. In this case you need to give it a specific gadget tree, but that's all doable.

[Andy Whitcroft (apw)]

Hello Łukasz, or anyone else affected,

Accepted livecd-rootfs into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.478.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Artful verification (package version 2.478.1):

$ cp -a `dpkg -L livecd-rootfs | grep "auto$"` .
$ sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=artful ARCH=amd64 lb config
[2017-11-15 11:29:05] lb_config
P: Executing auto/config script.
Building on amatsu
[2017-11-15 11:29:05] lb_config noauto --mode ubuntu --distribution artful --bootstrap-keyring ubuntu-keyring --binary-images none --memtest none --source false --build-with-chroot false --apt-source-archives false --linux-flavours virtual --initsystem none --bootloader none --initramfs-compression lzma --chroot-filesystem none --initramfs=none --system=normal --hdd-label=cloudimg-rootfs --ext-resize-blocks=536870912 --ext-block-size=4096 --ext-fudge-factor=15
P: Considering defaults defined in /etc/live/build.conf
P: Updating config tree for a ubuntu/amd64 system
$ sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=artful ARCH=amd64 lb build
(...)
$ ls chroot/
bin dev home lib media opt root sbin srv tmp var
boot etc initrd.img lib64 mnt proc run snap sys usr vmlinuz

The live-build succeeded and resulted in the rootfs in the chroot/ directory (without any final images) as expected.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Murray (brian-murray)]

Hello Łukasz, or anyone else affected,

Accepted livecd-rootfs into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.441.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Łukasz, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.478.1

---------------
livecd-rootfs (2.478.1) artful; urgency=medium

  [ Gary Wang ]
  * Add the IMAGEFORMAT=none to support generating a single rootfs
    (LP: #1730642)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 08 Nov 2017 12:02:04 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Ok, I noticed an issue with Gary's branch right now. I didn't see this during artful verification as the verification steps are missing one check and in artful the whole lb build has passed successfully. But on zesty I noticed that lb build failed on running a *binary* hook. All binary hooks should have been removed due to the changes, but they're not!

The reason is that both Gary and Steve didn't see that actually then we set IMAGEFORMAT=none, for PROJECT=ubuntu-cpc in auto/config IMAGEFORMAT is hard-coded back to ext4. This should not happen. Sadly I'll have to fail this update...

[Łukasz Zemczak (sil2100)]

I know now why it all worked for artful and bionic - there is a conditional that preserves the IMAGEFORMAT for artful+ which is missing from the earlier versions. Backporting that and re-releasing.

[Steve Langasek (vorlon)]

Hello Łukasz, or anyone else affected,

Accepted livecd-rootfs into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.441.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Łukasz, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.24 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Verification zesty (package version 2.441.10):

$ cp -a `dpkg -L livecd-rootfs | grep "auto$"` .
$ sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=zesty lb config
[2017-11-21 22:40:28] lb_config
P: Executing auto/config script.
Building on zesty-test.canonistack
[2017-11-21 10:40:28] lb_config noauto --mode ubuntu --distribution zesty --bootstrap-keyring ubuntu-keyring --binary-images none --memtest none --source false --build-with-chroot false --apt-source-archives false --linux-flavours virtual --initsystem none --bootloader none --initramfs-compression lzma --initramfs none --chroot-filesystem ext4 --initramfs=none --system=normal --hdd-label=cloudimg-rootfs --ext-resize-blocks=536870912 --ext-block-size=4096 --ext-fudge-factor=15
P: Considering defaults defined in /etc/live/build.conf
P: Updating config tree for a ubuntu/amd64 system
$ sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=zesty lb build
(...)
$ ls chroot/
bin dev home lib media opt root sbin srv tmp var
boot etc initrd.img lib64 mnt proc run snap sys usr vmlinuz
$ ls config/hooks/ | grep binary
$

[Łukasz Zemczak (sil2100)]

Verification xenial (package version 2.408.24):

$ cp -a `dpkg -L livecd-rootfs | grep "auto$"` .
$ sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=xenial lb config
[2017-11-21 22:00:52] lb_config
P: Executing auto/config script.
Building on xenial-test.canonistack
[2017-11-21 22:00:52] lb_config noauto --mode ubuntu --distribution xenial --bootstrap-keyring ubuntu-keyring --binary-images none --memtest none --source false --build-with-chroot false --apt-source-archives false --linux-flavours virtual --initsystem none --bootloader none --initramfs-compression lzma --cache false --chroot-filesystem none --initramfs=none --system=normal --hdd-label=cloudimg-rootfs --ext-resize-blocks=536870912 --ext-block-size=4096 --ext-fudge-factor=15
P: Considering defaults defined in /etc/live/build.conf
P: Updating config tree for a ubuntu/amd64 system
$ sudo PROJECT="ubuntu-cpc" ARCH=amd64 IMAGEFORMAT=none SUITE=xenial lb config
(...)
$ ls chroot/
bin dev home lib media opt root sbin srv tmp var
boot etc initrd.img lib64 mnt proc run snap sys usr vmlinuz
$ ls config/hooks/ | grep binary
$

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.408.24

---------------
livecd-rootfs (2.408.24) xenial; urgency=medium

  * For the IMAGEFORMAT=none parts to work for ubuntu-image classic builds, we
    need to make sure IMAGEFORMAT is preserved and not overwritten if defined.
    Backporting from artful.

livecd-rootfs (2.408.23) xenial; urgency=medium

  [ Gary Wang ]
  * Add the IMAGEFORMAT=none to support generating a single rootfs
    (LP: #1730642)

  [ Christopher Glass (Ubuntu) ]
  * Backport "minimized round 2" changes from trunk to Xenial (LP: #1731492)
  * Remove apt, debconf, dpkg cruft files from /var/cache and /var/lib in
    all our livefses.
  * Pass --cache false to lb config; otherwise we copy around caches of
    .debs that are never used properly, and which prevent us from emptying
    /var/cache/apt in images.
  * When building minimized cloud images, remove various packages that we
    don't want installed by default. Some are tools that aren't needed for
    non-interactive use; some are libraries whose reverse-dependencies
    will have already been removed; and one, open-vm-tools, should only be
    included in images that are targeted to VMWare (which is not the case
    for any of the current minimal images), rather than being included
    directly in the cloud-image seed.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 21 Nov 2017 17:01:14 +0100

[Łukasz Zemczak (sil2100)]

Quickly looking at the s390x livecd-rootfs failure, it doesn't seem to be related to any of the changes that have been made by the new version. It seems that during bootstrapping, udev is failing to configure properly. I'll ask xnox to maybe take a look at it and see what could be the reason. That being said, I'd opt for adding a hint on this as it's certainly unrelated.

[Łukasz Zemczak (sil2100)]

13:11 < xnox> sil2100, it has never passed.
13:11 < xnox> sil2100, "default-bootstraps SKIP Test requires machine-level isolation but testbed does not provide that" it used "always-skip", now it is in fact "always failed"
13:12 < xnox> sil2100, the reason why it fails on zesty, has been fixed but has not yet been SRUed. =/
13:12 < xnox> sil2100, i will try to SRU that fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.441.10

---------------
livecd-rootfs (2.441.10) zesty; urgency=medium

  * For the IMAGEFORMAT=none parts to work for ubuntu-image classic builds, we
    need to make sure IMAGEFORMAT is preserved and not overwritten if defined.
    Backporting from artful.

livecd-rootfs (2.441.9) zesty; urgency=medium

  [ Gary Wang ]
  * Add the IMAGEFORMAT=none to support generating a single rootfs
    (LP: #1730642)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 21 Nov 2017 16:56:58 +0100