arm64 cloud image contains flash-kernel which leads to apt errors on upgrade

[Stephan Fabel]

To reproduce on ARM:

$ lxc launch ubuntu:18.04 container
$ lxc shell container
# apt update
# apt upgrade

Will return error due to presence of the flash-kernel package.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

in livecd-rootfs:

ubuntu-cpc)
   case $ARCH in
        armhf)
              KERNEL_FLAVOURS=generic-lpae
              add_package install flash-kernel
              ;;
        arm64)
              add_package install flash-kernel
              ;;
   esac

So CPC team should stop including flash-kernel (which is a bootloader) in all squashfs, and only install it images where you install bootloaders.
Just like grub, it shouldn't be in the LXD image.

[Patricia Gaughen (gaughen)]

Dimitri - we can look to do that for future, but that doesn't address the upgrade issue. Did something change that made this break?

[Steve Langasek (vorlon)]

The flash-kernel package shouldn't be included in the squashfs, it only belongs in the cloud images. Nevertheless, because it's there, we need to fix the flash-kernel maintainer scripts to not fail on existing images.

[Dave Jones (waveform)]

> Dimitri - we can look to do that for future, but that doesn't address the upgrade issue. Did something change that made this break?

Almost certainly my change to ensure flash-kernel always runs on upgrade (as it's the subject of a trigger, and trigger invocations are lost on package upgrade which results in all sorts of fun with u-boot and the migration of boot configurations on pi for the .4 release).

I'll add a check to ensure flash-kernel dies quietly in the case it finds itself in a container.

[Dave Jones (waveform)]

Some test packages for focal and bionic are available from the following PPA:

https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel/+packages

The relevant branches are:

https://code.launchpad.net/~waveform/ubuntu/+source/flash-kernel/+git/flash-kernel/+ref/no-lxc-focal

https://code.launchpad.net/~waveform/ubuntu/+source/flash-kernel/+git/flash-kernel/+ref/no-lxc-bionic

In the focal version I've also taken the liberty of excising FLASH_KERNEL_SKIP as it's no longer used by anything in focal (it's only used by the ac100 platform in bionic), which reduces our delta with upstream. In both versions, the test for containers is moved from the initramfs and kernel hooks into flash-kernel itself. This should fix the upgrade issue, but do let me know if any other issues crop up in testing!

[Robert C Jennings (rcj)]

I have a questionable recreate and maybe Stephan Fabel can confirm if this is what he's seeing.

I have launched an arm64 VM on AWS and on that VM I have started an 18.04 container from the 'ubuntu:' remote. The container has a build serial of 20190307.
Running 'apt update; apt upgrade -y' from this older image is enough to find an update for flash-kernel (3.98ubuntu11~18.04.1). During the upgrade I see the following message:
  flash-kernel: /boot/vmlinuz- or /boot/initrd.img- missing; expecting future run
But the exit code from the upgrade is 0 and there are no broken packages.

Stephan, is this the extent of the failure that you have seen? I was going to help validate the fix from Dave but I want to ensure I'm seeing the same failure first.

[Robert C Jennings (rcj)]

For others that would like to attempt a recreate, the image I'm using is:

$ lxc image info ubuntu:08e14ca27965
Fingerprint: 08e14ca2796594a0a52f62a1974774db6f4ecfbb2a3da7f3a5c1e2e0abff7362
Size: 161.48MB
Architecture: aarch64
Public: yes
Timestamps:
    Created: 2019/03/07 00:00 UTC
    Uploaded: 2019/03/07 00:00 UTC
    Expires: 2023/04/26 00:00 UTC
    Last used: never
Properties:
    architecture: arm64
    label: release
    serial: 20190307
    description: ubuntu 18.04 LTS arm64 (release) (20190307)
    os: ubuntu
    release: bionic
    version: 18.04

[Robert C Jennings (rcj)]

If that is the issue that Stephan is seeing then I have not seen the issue with the bionic package in Dave's PPA:

$ lxc launch ubuntu:08e14ca27965 b
$ lxc shell b
 $ apt-add-repository ppa:waveform/flash-kernel
 $ apt install -y flash-kernel

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  flash-kernel
1 upgraded, 0 newly installed, 0 to remove and 182 not upgraded.
Need to get 62.0 kB of archives.
After this operation, 11.3 kB of additional disk space will be used.
Get:1 http://ppa.launchpad.net/waveform/flash-kernel/ubuntu bionic/main arm64 flash-kernel arm64 3.98ubuntu11~18.04.2~ppa1 [62.0 kB]
Fetched 62.0 kB in 1s (84.8 kB/s)
Preconfiguring packages ...
(Reading database ... 28535 files and directories currently installed.)
Preparing to unpack .../flash-kernel_3.98ubuntu11~18.04.2~ppa1_arm64.deb ...
Unpacking flash-kernel (3.98ubuntu11~18.04.2~ppa1) over (3.90ubuntu3.18.04.2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up flash-kernel (3.98ubuntu11~18.04.2~ppa1) ...
Installing new version of config file /etc/flash-kernel/bootscript/bootscr.uboot-generic ...
Installing new version of config file /etc/initramfs/post-update.d/flash-kernel ...
Installing new version of config file /etc/kernel/postinst.d/zz-flash-kernel ...
Installing new version of config file /etc/kernel/postrm.d/zz-flash-kernel ...

[Robert C Jennings (rcj)]

Dave, using the latest focal daily lxd container (serial 20200313) I can remove and install flash-kernel to see the "flash-kernel: /boot/vmlinuz- or /boot/initrd.img- missing; expecting future run" failure and confirm that installing from the ppa (3.98ubuntu12~ppa1) gives the same output as above for my bionic test. Still, I'd like Stephan to confirm that this was the error he was seeing when he filed the case.

[Stephan Fabel (sfabel)]

Hey - so I tried again today, just following the same steps:

-------snip--------
ubuntu@rpi4:~$ lxc launch ubuntu:18.04
Creating the instance
Instance name is: funky-whale
Starting funky-whale
ubuntu@rpi4:~$ lxc shell funky-whale
mesg: ttyname failed: No such device
root@funky-whale:~# apt update && apt -y upgrade
Hit:1 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Get:2 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease [88.7
kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease [74.6
kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease [88.7
kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 Packages
[8316 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports bionic/universe Translation-en
[4941 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports bionic/multiverse arm64 Packages
[126 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports bionic/multiverse Translation-en
[108 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64
Packages [637 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main
Translation-en [306 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports bionic-updates/restricted
Translation-en [9020 B]
Get:12 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64
Packages [944 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe
Translation-en [327 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports bionic-updates/multiverse arm64
Packages [3304 B]
Get:15 http://ports.ubuntu.com/ubuntu-ports bionic-updates/multiverse
Translation-en [4636 B]
Get:16 http://ports.ubuntu.com/ubuntu-ports bionic-backports/main arm64
Packages [2512 B]
Get:17 http://ports.ubuntu.com/ubuntu-ports bionic-backports/main
Translation-en [1644 B]
Get:18 http://ports.ubuntu.com/ubuntu-ports bionic-backports/universe arm64
Packages [4016 B]
Setting up gcc-8-base:arm64 (8.3.0-26ubuntu1~18.04) ...
(Reading database ... 28977 files and directories currently installed.)
Preparing to unpack .../libstdc++6_8.3.0-26ubuntu1~18.04_arm64.deb ...
Unpacking libstdc++6:arm64 (8.3.0-26ubuntu1~18.04) over
(8.3.0-6ubuntu1~18.04.1) ...
Setting up libstdc++6:arm64 (8.3.0-26ubuntu1~18.04) ...
(Reading database ... 28977 files and directories currently installed.)
Preparing to unpack .../libgcc1_1%3a8.3.0-26ubuntu1~18.04_arm64.deb ...
Unpacking libgcc1:arm64 (1:8.3.0-26ubuntu1~18.04) over
(1:8.3.0-6ubuntu1~18.04.1) ...
Setting up libgcc1:arm64 (1:8.3.0-26ubuntu1~18.04) ...
(Reading database ... 28977 files and directories currently installed.)
Preparing to unpack .../libsqlite3-0_3.22.0-1ubuntu0.3_arm64.deb ...
Unpacking libsqlite3-0:arm64 (3.22.0-1ubuntu0.3) over (3.22.0-1ubuntu0.2)
...
Preparing to unpack .../rsync_3.1.2-2.1ubuntu1.1_arm64.deb ...
Unpacking rsync (3.1.2-2.1ubuntu1.1) over (3.1.2-2.1ubuntu1) ...
Preparing to unpack .../landscape-common_18.01-0ubuntu3.5_arm64.deb ...
Unpacking landscape-common (18.01-0ubuntu3.5) over (18.01-0ubuntu3.4) ...
Preparing to unpack .../sosreport_3.9-1ubuntu0.18.04.1_arm64.deb ...
Unpacking sosreport (3.9-1ubuntu0.18.04.1) over (3.6-1ubuntu0.18.04.4) ...
Preparing to unpack .../unattended-upgrades_1.1ubuntu1.18.04.14_all.deb...