flash-kernel 3.106ubuntu13 causes a trigger cycle when installing arm64 kernels on VMs

[Francis Ginther]

We're seeing the following error when trying to install kernel packages on cloud instances with the latest flash-kernel, 3.106ubuntu13.

sudo apt-get install --yes linux-generic
...
: /boot/vmlinuz is now a symlink to vmlinuz-6.1.0-16-generic
I: /boot/initrd.img is now a symlink to initrd.img-6.1.0-16-generic
Setting up linux-headers-generic (6.1.0.16.16) ...
Setting up linux-image-generic (6.1.0.16.16) ...
Setting up linux-generic (6.1.0.16.16) ...
Processing triggers for man-db (2.11.2-1) ...
Processing triggers for flash-kernel (3.106ubuntu13) ...
flash-kernel: requested kernel 6.1.0-16-generic is in state triggers-pending.
flash-kernel: deferring update (trigger activated)
dpkg: cycle found while processing triggers:
 chain of packages whose triggers are or may be responsible:
  flash-kernel -> linux-image-6.1.0-16-generic
 packages' pending triggers which are or may be unresolvable:
  linux-image-6.1.0-16-generic: linux-update-6.1.0-16-generic
  flash-kernel: flash-kernel
dpkg: error processing package linux-image-6.1.0-16-generic (--configure):
 triggers looping, abandoned
Processing triggers for flash-kernel (3.106ubuntu13) ...
flash-kernel: requested kernel 6.1.0-16-generic is in state half-configured.
flash-kernel: deferring update (trigger activated)
Processing triggers for flash-kernel (3.106ubuntu13) ...
flash-kernel: requested kernel 6.1.0-16-generic is in state half-configured.
flash-kernel: deferring update (trigger activated)
Processing triggers for flash-kernel (3.106ubuntu13) ...
flash-kernel: requested kernel 6.1.0-16-generic is in state half-configured.
flash-kernel: deferring update (trigger activated)
dpkg: cycle found while processing triggers:
 chain of packages whose triggers are or may be responsible:
  flash-kernel -> flash-kernel -> flash-kernel -> flash-kernel
 packages' pending triggers which are or may be unresolvable:
  flash-kernel: flash-kernel
dpkg: error processing package flash-kernel (--configure):
 triggers looping, abandoned
Errors were encountered while processing:
 linux-image-6.1.0-16-generic
 flash-kernel
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

I was able to reproduce this on an aws image with serial 20230306, but not on one with serial 20230303 (which was using flash-kernel 3.106ubuntu12. I also see this on the latest google cloud images.

[Dave Jones (waveform)]

I had a nasty feeling this might rear its head on some platform. The question in this case is: why is the kernel stuck in half-configured state?

The new flash-kernel test is expecting that to resolve to "installed" before it can proceed because otherwise it risks flashing the kernel, but failing to flash the initrd (or flashing an invalid initrd).

In fact, that looks like the correct decision here because I don't see the initramfs-tools trigger firing to generate the initrd? The line at the top ("I: /boot/initrd.img is now a symlink to initrd.img-6.1.0-16-generic") doesn't mean the initrd exists, just that the symlink has been created and at this stage, at least on Xilinx and Pi images, that means the symlink is dead and definitely should not be flashed; once the kernel package is fully "installed", we can be certain the initrd exists and is correct.

[Francis Ginther (fginther)]

Install log with an AWS image from 20230305 using flash-kernel 3.106ubuntu12.

[Francis Ginther (fginther)]

Install log with an AWS image from 20230315 using flash-kernel 3.106ubuntu13.

[Dave Jones (waveform)]

Thanks for the logs -- the latter is most interesting in that it indicates the eventual cycle *isn't* the first cycle error that occurs. One occurs earlier on flash-kernel's second execution, when the kernel is reported as being in "triggers-pending" state (presumably because its initrd hasn't been generated yet).

Because dpkg decides (for some reason I don't understand yet) that f-k has triggered a cycle *there*, it places the kernel in half-configured state (because the triggers it's awaiting have failed), and that explains the later cycle that gets triggered.

So, the question becomes: why is dpkg deciding that f-k has triggered a cycle that early when it doesn't in the Xilinx and Pi cases... Is there something different about this kernel to others? In particular, with regard to its initrd and corresponding initramfs-tools trigger?

In the meantime, I'll have a dig into the dpkg cycle detection and try and figure out why that first cycle is detected when it doesn't in the Xilinx and Pi cases.

[Dave Jones (waveform)]

I've uploaded a tentative fix for this to ppa:waveform/flash-kernel (https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel/+packages). I've tested this locally on several Pi images and found it operates correctly so far, but then so did the prior fix.

@fginther could you test this version in at least one of the arm64 clouds in question?

[Francis Ginther (fginther)]

@waveform, yes, I can start testing this now.

[Francis Ginther (fginther)]

@waveform,

I've tested on both aws and gcp, by installing the flash-kernel update first, then installing the linux-generic kernel. Here's the apt-get output:

$ apt-get install --yes kernel-testing--linux--full--generic
Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
  libnuma1
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  firmware-sof-signed linux-firmware linux-headers-6.2.0-18
  linux-headers-6.2.0-18-generic linux-headers-generic
  linux-image-6.2.0-18-generic linux-image-generic
  linux-modules-6.2.0-18-generic linux-modules-extra-6.2.0-18-generic
  wireless-regdb
Suggested packages:
  fdutils linux-doc | linux-source-6.2.0 linux-tools
The following NEW packages will be installed:
  firmware-sof-signed linux-firmware linux-generic linux-headers-6.2.0-18
  linux-headers-6.2.0-18-generic linux-headers-generic
  linux-image-6.2.0-18-generic linux-image-generic
  linux-modules-6.2.0-18-generic linux-modules-extra-6.2.0-18-generic
  wireless-regdb
0 upgraded, 11 newly installed, 0 to remove and 5 not upgraded.
Need to get 428 MB of archives.
After this operation, 1821 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 firmware-sof-signed all 2.2.4-1 [651 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-firmware all 20230323.gitbcdcfbcf-0ubuntu1 [281 MB]
Get:3 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-modules-6.2.0-18-generic arm64 6.2.0-18.18 [28.0 MB]
Get:4 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-image-6.2.0-18-generic arm64 6.2.0-18.18 [20.0 MB]
Get:5 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 wireless-regdb all 2022.06.06-0ubuntu1 [10.3 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-modules-extra-6.2.0-18-generic arm64 6.2.0-18.18 [82.0 MB]
Get:7 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-image-generic arm64 6.2.0.18.18 [7936 B]
Get:8 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-headers-6.2.0-18 all 6.2.0-18.18 [13.1 MB]
Get:9 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-headers-6.2.0-18-generic arm64 6.2.0-18.18 [2924 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-headers-generic arm64 6.2.0.18.18 [7844 B]
Get:11 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 linux-generic arm64 6.2.0.18.18 [1704 B]
Fetched 428 MB in 5s (78.2 MB/s)
Selecting previously unselected package firmware-sof-signed.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 68792 files and d...

[Launchpad Janitor (janitor)]

This bug was fixed in the package flash-kernel - 3.106ubuntu14

---------------
flash-kernel (3.106ubuntu14) lunar; urgency=medium

  * Handle "not fully installed" kernels differently by quitting instead of
    re-triggering to avoid a dpkg trigger cycle (LP: #2011925)

 -- Dave Jones <email address hidden> Thu, 23 Mar 2023 14:12:19 +0000