Restore the ability to produce core16 images

[Łukasz Zemczak]

[Impact]

Ubuntu Core is an all-snap Ubuntu variant that we are building through cdimage. In the past there was only one core series, 16, and all the images we were building were based on that series. Thanks to that all the support for building ubuntu-core images in livecd-rootfs through ubuntu-image was only needed in later series, as all were build using devel. Currently we have two different ubuntu-core image sets based of different bases: core16 and core18 images. As a convention, we have decided that cdimage (and or other components) will be building ubuntu-core images based of the suite that the build is being run for. Since core16 is based on xenial and core18 on bionic, we have decided the following scheme:

 * Suite: xenial -> ubuntu-core 16 image
 * Suite: bionic or later -> ubuntu-core 18 image

For this to work, we need to have all the needed changes in xenial and bionic. livecd-rootfs on xenial is missing all the ubuntu-image build-enablement changes, so these need to be backported. Besides that, both xenial and bionic need to get backports of the per-distro-series core-series selection.

Without these fixes in xenial, no ubuntu-core 16 images can be built anymore. This is needed in case we'd have need for any security-vulnerability fix image re-releases.

[Test Case]

For xenial:

 * On a xenial system install latest xenial-proposed livecd-rootfs
 * Create a temporary directory and cd into it
 * Copy the /usr/share/livecd-rootfs/live-build/auto directory into the current directory
 * sudo PROJECT=ubuntu-core SUITE=xenial ARCH=amd64 PROPOSED=0 IMAGEFORMAT="ubuntu-image" lb config
   -> Make sure that the configured model assertion is for core16
 * sudo PROJECT=ubuntu-core SUITE=xenial ARCH=amd64 PROPOSED=0 IMAGEFORMAT="ubuntu-image" lb build
   -> Make sure the build runs and finishes correctly
 * Boot test the newly created image with kvm
   -> Make sure the image is still bootable

Also, since we are basically starting to use ubuntu-image with this xenial upload of livecd-rootfs (which was in universe for this series), we need to make sure the package is still installable with just main enabled.

 * On a clean chroot (without livecd-rootfs or any of its dependencies installed)
 * Modify sources.list to only have main enabled
 * sudo apt update
 * sudo apt install livecd-rootfs
   -> Make sure the package and its dependencies install correctly

[Regression Potential]

As part of the backport only new code is added - code that is executed in certain code paths only. Most regressions might appear in the ubuntu-core image generation, which was not available for xenial. But in case the code has been added in the wrong way, potentially this might result in livecd-rootfs not being able to generate proper classic images anymore. A run of lb config and lb build for a non ubuntu-image project is recommended.

[Łukasz Zemczak (sil2100)]

The xenial SRU will require an MIR for ubuntu-image, as it's still in universe in that suite.

A lot of build-dependencies of ubuntu-image are in universe, but since those are only build-deps it should not matter. As for binary dependencies, we have these:

 * python3-parted binary and source package is in universe
 * python3-voluptuous is in universe, but its source voluptuous is already in main; file an ubuntu-archive bug for promoting the current preferred alternative

The python3-voluptuous binary needs to be promoted to main and we'll have to fill in an MIR of pyparted. pyparted has no universe dependencies and has already been promoted to main in all the later series.

[Steve Langasek (vorlon)]

I wasn't able to find the bug report for python3-voluptuous, but I've promoted this now.

[Łukasz Zemczak (sil2100)]

I promoted both pyparted and ubuntu-image to main on xenial right now.

[Łukasz Zemczak (sil2100)]

This is now uploaded to xenial. Please note that some of the changes have not yet been released into bionic - but that's intended. I would like to request that the xenial livecd-rootfs of this SRU land first, before we prepare bionic. The reason for that is that currently, thanks to bionic missing these changes, we can use this series to temporarily build core16 images. Otherwise we'd have no easy way of doing that before the xenial livecd-rootfs fully migrates into updates (and right now that's still needed).

[Brian Murray (brian-murray)]

Hello Łukasz, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.41 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robert C Jennings (rcj)]

I have tested on amd64 with livecd-rootfs 2.408.41 from -proposed following the test case in the description. My testing diverged from the test plan in that I am driving livecd-rootfs with ubuntu-old-fashioned[1] which builds with launchpad-buildd tools in a lxd container to replicate the buildd environment. In this environment the build fails due to the lack of 'sudo':

[2018-12-10 18:06:08] lb_build
P: Executing auto/build script.
Fetching core
Fetching pc-kernel
Fetching pc
Warning: for backwards compatibility, `ubuntu-image` falls back to `ubuntu-image snap` if no subcommand is given
-o/--output is deprecated; use -O/--output-dir instead
Crash in state machine
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ubuntu_image/__main__.py", line 341, in main
    list(state_machine)
  File "/usr/lib/python3/dist-packages/ubuntu_image/state.py", line 82, in __next__
    step()
  File "/usr/lib/python3/dist-packages/ubuntu_image/common_builder.py", line 337, in populate_filesystems
    self._populate_one_volume(name, volume)
  File "/usr/lib/python3/dist-packages/ubuntu_image/common_builder.py", line 290, in _populate_one_volume
    preserve_ownership=True)
  File "/usr/lib/python3/dist-packages/ubuntu_image/helpers.py", line 233, in mkfs_ext4
    with mount(img_file) as mountpoint:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/ubuntu_image/helpers.py", line 207, in mount
    run('sudo mount -oloop {} {}'.format(img, mountpoint))
  File "/usr/lib/python3/dist-packages/ubuntu_image/helpers.py", line 115, in run
    **args)
  File "/usr/lib/python3.5/subprocess.py", line 693, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'sudo'
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_livefs.py", line 161, in run
    self.build()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_livefs.py", line 152, in build
    self.run_build_command(["lb", "build"], env=base_lb_env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_livefs.py", line 79, in run_build_command
    return self.backend.run(args, cwd="/build", **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 490, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-xenial-amd64', '--env', 'PROJECT=ubuntu-core', '--env', 'ARCH=amd64', '--', '/bin/sh', '-c', 'cd /build && linux64 lb build']' returned non-zero exit status 1

The rootfs used by the builders does not have 'sudo' installed, it should be a package dependency for livecd-rootfs based on this testing. I'm going to mark this as verification-failed-xenial. However, I did install 'sudo' m...

[Robert C Jennings (rcj)]

I said, "it should be a package dependency for livecd-rootfs based on this testing" but now that I look at the traceback it's the case that python3-ubuntu-image is using 'sudo' but does not depend on it. So I'd say that maybe this SRU can be marked verification-passed but ubuntu-image source needs a bug files to add 'sudo' to its dependencies.

[Robert C Jennings (rcj)]

New bug #1807750 filed against ubuntu-image for the sudo dependency. Marking this as verification-done to unblock other SRUs and because the change introduced by this SRU does allow for building of core16 images using the steps outlined in the test case (and in the buildds once ubuntu-image is fixed).

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.408.41

---------------
livecd-rootfs (2.408.41) xenial; urgency=medium

  * Disable checksum generation (LP: #1799773)

livecd-rootfs (2.408.40) xenial; urgency=medium

  [ Cody Shepherd ]
  * Increasing modularity in relocation of /etc/resolv.conf to enable users
    of live-build/functions to manipulate /etc/resolv.conf in a more
    granular and future-proof way. LP: #1801134.

livecd-rootfs (2.408.39) xenial; urgency=medium

  * Backport support for building ubuntu-core images with ubuntu-image (using
    IMAGEFORMAT=ubuntu-image).
  * Decide what model assertion series to fetch depending on the suite. Use 16
    for xenial and 18 for other series (bionic+). This enables core18 image
    builds (LP: #1799736).

 -- Robert C Jennings <email address hidden> Fri, 16 Nov 2018 08:37:57 -0800

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.