[SRU] flash-kernel failure when upgrading f-k and kernel in the same cycle

Bug #2007827 reported by ethan.hsieh
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
flash-kernel (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Confirmed
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
Kinetic
Invalid
Undecided
Unassigned
Lunar
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
In version 3.104ubuntu15 of flash-kernel, when both f-k and the kernel are upgraded in the same cycle, depending on the ordering of dpkg trigger execution, f-k may find the content of /boot "inconsistent" causing it to fail and return error exit status 1.

Erorr message:
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for flash-kernel (3.104ubuntu15) ...
flash-kernel: installing version 5.15.0-1018-xilinx-zynqmp
Initrd required for FIT method
dpkg: error processing package flash-kernel (--configure):
 installed flash-kernel package post-installation script subprocess returned error exit status 1
Processing triggers for linux-image-5.15.0-1018-xilinx-zynqmp (5.15.0-1018.20) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.15.0-1018-xilinx-zynqmp

flash-kernel gets the latest kernel version by "linux-version list".
When flash-kernel was triggered to generate fitimage, the kernel version is "5.15.0-1018" and the initrd for it wasn't ready. So, flash-kernel failed to generate the fitimage.

A subsequent run of "apt install -f" fixed things because, by that point, the kernel's own trigger had executed, ensuring that update-initramfs had been run. In the case that f-k is run "prematurely" and finds itself in this situation (/boot/kernel-$[ver} exists, but /boot/initrd-${ver}) doesn't), it should probably bail out silently under the assumption that whatever is responsible for it will rectify the situation and trigger f-k again (as happens in the kernel postinst hooks).

[Test Case]
1. Flash an old image (with an out of date kernel and flash-kernel)
2. sudo apt-get update
3. sudo apt install flash-kernel with the fix and linux packages
4. Upgrade should proceed without issue

[Regression Potential]
As with the previous flash-kernel uploads, it is possible that a breakage in the changed code can lead to issues with upgrading kernels (due to f-k being executed via a trigger at the end) or with Xilinx devices in the field not upgrading correctly. I will test all the changes extensively though.

Related issues:
LP: #1861292 flash-kernel failure during kernel upgrade

Revision history for this message
ethan.hsieh (ethan.hsieh) wrote :

The same issue (LP: #1861292) was fixed on focal, but the fix is not available after jammy. The patch for focal cannot be applied to lunar because of [1].

---
[1] https://salsa.debian.org/installer-team/flash-kernel/-/commit/30f64ebd28553e74986aa4a80963b42f478a4d3d

description: updated
tags: added: oem-priority originate-from-2003041
Revision history for this message
ethan.hsieh (ethan.hsieh) wrote :
Revision history for this message
ethan.hsieh (ethan.hsieh) wrote :

Verified the test flash-kernel package on a Xilinx ZCU board. With the test package, fitimage can be generated without any problem.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "flash-kernel-lunar.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Dave Jones (waveform) wrote :
Download full text (4.1 KiB)

Hmm, this is a tricky one and there's a lot I don't like here.

The first thing I don't like is I probably caused this issue back in focal :). I was fixing an issue of flash-kernel triggers getting "forgotten" during flash-kernel updates (LP: #1667742).

The second thing I don't like is that we apparently haven't noticed this (quite serious) issue in all the releases since. That's ... suspicious? I would at least have expected this to have cropped up for some people on the pi images which ought to be seeing something similar and are quite widely used.

The third thing I'm not particularly keen on is the proposed fix: it causes flash-kernel to exit *assuming* it'll be run in future without actually guaranteeing that (contrast with the trigger deferral logic near the top of main where flash-kernel triggers itself and then exits to ensure it will definitely be run later). It also doesn't necessarily account for the scenario where flash-kernel is being upgraded to fix an issue in flash-kernel's historic behaviour. If the current kernel needs re-installing anyway (because there was some defect in the way an old version of f-k handled things), even when a new kernel version will be installed later, that should still be done as we can't guarantee that the new kernel's initrd will be generated successfully.

Hence, the two things we really want to avoid:

* Exiting without error when an error has actually occurred (i.e. the initrd is missing, but it's *actually* missing not just "waiting to be generated")

* Guaranteeing we don't "skip" flash-kernel executions when flash-kernel itself is being upgraded, even when they might *seem* redundant (they're only redundant in the case that everything works correctly, but the error scenarios are valid edge cases)

I've spent a few hours digging into all the trigger logic that exists between the kernel images, initramfs-tools (responsible for generating the initrd), and flash-kernel. I'm reasonably convinced at this point that we can't *prevent* flash-kernel from running at a point where a new kernel isn't *completely* installed. The fact that flash-kernel *must* run when it itself is upgraded means we're always vulnerable to being run at that point.

However, flash-kernel is being run to install "the lastest version". It uses "linux-version list" to get the installed kernels. That operation lists the kernels which *exist* (as, say, /boot/vmlinuz-5.15.0-1018-xilinx-zynqmp) under /boot, but doesn't care whether the kernel package is actually *installed fully* by dpkg yet.

For example: /boot/vmlinuz-5.15.0-1018-xilinx-zynqmp exists because the linux-image-5.15.0-1018-xilinx-zynqmp has been unpacked, but it must still be in "triggers-awaiting" state because the initramfs-tools trigger has not yet run to generate the corresponding initrd.

So ... I think the root of the issue here is that flash-kernel is not being sufficiently discriminating of its selection of the "latest" kernel; it considers not-fully-installed kernels to be valid for installation. In fact, even if the initrd *does* exist, if triggers are still pending it still shouldn't consider that a valid candidate (this can occur if, for example, a linux-m...

Read more...

Revision history for this message
Dave Jones (waveform) wrote :
Download full text (6.3 KiB)

Okay, I've now confirmed that this is indeed the rather more wide-ranging bug that I suspected it was (it also affects the Pi images, but in a different way). I'll try and explain the problem with reference to the log provided in comment:3. First, the scenario:

* the device currently has kernel 5.15.0-1015-xilinx-zynqmp
* kernel 5.15.0-1018-xilinx-zynqmp is available for upgrade
* the device currently has flash-kernel 3.104ubuntu14 installed
* flash-kernel 3.106ubuntu12test1 is available for upgrade

This in turn means that the following package states are currently true:

* linux-image-5.15.0-1015-xilinx-zynqmp is "installed"
* linux-image-5.15.0-1018-xilinx-zynqmp is "not installed"
* flash-kernel is "installed"

Now selected entries from the log, in chronological order:

  Preparing to unpack .../0-flash-kernel_3.106ubuntu12test1_arm64.deb ...
  Unpacking flash-kernel (3.106ubuntu12test1) over (3.104ubuntu14) ...
  ...
  Selecting previously unselected package linux-image-5.15.0-1018-xilinx-zynqmp.
  Preparing to unpack .../4-linux-image-5.15.0-1018-xilinx-zynqmp_5.15.0-1018.20_arm64.deb ...
  ...
  Setting up flash-kernel (3.106ubuntu12test1) ...

At this point:

* flash-kernel has now been upgraded but is now in "triggers-pending" as its upgrade triggers itself (this is the issue fixed in LP: #1667742)
* linux-image-5.15.0-1015-xilinx-zynqmp is still "installed"
* linux-image-5.15.0-1018-xilinx-zynqmp is now "unpacked"

Further entries:

  I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-1018-xilinx-zynqmp
  I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-1018-xilinx-zynqmp
  Setting up linux-image-xilinx-zynqmp (5.15.0.1018.19) ...
  Setting up linux-modules-5.15.0-1018-xilinx-zynqmp (5.15.0-1018.20) ...
  Setting up linux-xilinx-zynqmp (5.15.0.1018.19) ...

At this point, linux-image-5.15.0-1018-xilinx-zynqmp is now "triggers-awaiting". The kernel postinst has run, updating the /boot symlinks to point to the 1018 kernel. However, the /boot/initrd.img symlink is "dead" because the initrd for this version has not yet been generated. This doesn't matter though as the /boot kernel isn't the one actually used at boot time; it's just the one that flash-kernel copies to <wherever the platform needs>, and flash-kernel hasn't been called yet.

Carrying on:

  Processing triggers for initramfs-tools (0.140ubuntu13) ...
  initramfs-tools: update-initramfs -u
  update-initramfs: Generating /boot/initrd.img-5.15.0-1015-xilinx-zynqmp

The initramfs-tools hook is called for the *old* kernel, regenerating its initrd. This appears largely pointless (and in this case, actually is) but is necessary to cover the case of linux-modules-extra packages being added / removed, requiring a regeneration of the initrd (and possibly other scenarios I'm unaware of).

Carrying on:

  Processing triggers for flash-kernel (3.106ubuntu12test1) ...
  flash-kernel: installing version 5.15.0-1018-xilinx-zynqmp

Here's the root of the error. flash-kernel finally runs "properly" (no deferral), and uses "linux-version list" to determine the set of installed kernel versions. This simply looks for /boot/vmlinu[xz]-$ver files and reports the $ver suffixes. At this ...

Read more...

Revision history for this message
Dave Jones (waveform) wrote :

I've prepared a patched version of flash-kernel for lunar in ppa:waveform/flash-kernel (version 3.106ubuntu13~ppa3 at https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel), with test coverage of the new functions. I've tested that this fixes the issue illustrated on the Pi above which I believe has the same root cause as on the Xilinx boards but I would be most grateful if the fix could be tested on the Xilinx platform too.

To test:

* Install a stock lunar preinstalled server image
* sudo add-apt-repository ppa:waveform/flash-kernel
* sudo apt install linux-xilinx-zynqmp flash-kernel
* Ensure that upgrade proceeds without error
* sudo reboot
* Ensure system boots with new kernel version (and initrd!) successfully

As noted, the fix will result in a "redundant" run of flash-kernel instead of the skip, but this is safe and actually desirable for certain scenarios. I'll convert this to an SRU if verification on lunar is successful (and will forward the patch to Debian subsequently).

Revision history for this message
ethan.hsieh (ethan.hsieh) wrote :

@Dave

I tried the test package[1]. It can generate a fitimage with old kernel without any problem. But, the fitimage for new kernel isn't generated after new initrd is ready.

Here is the log: (For detailed log, please see the attached file)
update-initramfs: Generating /boot/initrd.img-5.15.0-1018-xilinx-zynqmp
...
Ignoring old or unknown version 5.15.0-1018-xilinx-zynqmp (latest is 5.15.0-1015-xilinx-zynqmp)
/etc/kernel/postinst.d/zz-flash-kernel:
Ignoring old or unknown version 5.15.0-1018-xilinx-zynqmp (latest is 5.15.0-1015-xilinx-zynqmp)

---
[1] 3.106ubuntu13~ppa3, https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel

Revision history for this message
Dave Jones (waveform) wrote :

Okay, attempt number 3! After another long dive into dpkg states, triggers, and the interactions around the kernel and initrd I think I've finally got a version that should:

1. Never install a kernel which isn't fully installed (unless --force is specified)

2. Avoids the race on waiting for a kernel to become fully installed during an apt run (when it discovers it's being asked to install a non-fully-installed kernel, it defers itself by retriggering)

3. Avoids duplicated runs (it does use the full version list from "linux-version list", but defers in the case that the latest isn't fully installed)

It's currently building the aforementioned PPA (https://launchpad.net/~waveform/+archive/ubuntu/flash-kernel) and the test procedure should be the same:

* Install a stock lunar preinstalled server image
* sudo add-apt-repository ppa:waveform/flash-kernel
* sudo apt install linux-xilinx-zynqmp flash-kernel
* Ensure that upgrade proceeds without error
* sudo reboot
* Ensure system boots with new kernel version (and initrd!) successfully

I've tested this several times with the raspi images (that displayed a related, but different symptom) in circumstances that do replicate the race condition without the fix and things are successful there, so I'm hoping that's also the case on Xilinx.

Revision history for this message
ethan.hsieh (ethan.hsieh) wrote :

Hi Dave

With flash-kernel (3.106ubuntu13~ppa6), the issue is gone and a new fitimgae with new kernel is generated. Please see the attached file for detailed log.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package flash-kernel - 3.106ubuntu13

---------------
flash-kernel (3.106ubuntu13) lunar; urgency=medium

  * Ensure that only kernels in fully "installed" state are considered for
    flashing (LP: #2007827)
  * test_functions: Add tests for include_only_flavors
  * d/control: Remove redundant awk dependency to appease lintian
  * Bump standards version

 -- Dave Jones <email address hidden> Mon, 27 Feb 2023 11:53:49 +0000

Changed in flash-kernel (Ubuntu Lunar):
status: New → Fix Released
Revision history for this message
satmandu (satadru-umich) wrote :

I've been generating kernel images with this script from the raspberry pi foundation kernels for some time:

https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

flash-kernel gives me this error now in lunar:

Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
dpkg-query: no packages found matching

I've had to modify /var/lib/dpkg/info/flash-kernel.postinst to have "exit 0" at the end, and I've also modified the beginning to replace "set -e" with "set -x" to get some more info on what's happening.

Trying to install the latest generated image, I get this error:

Setting up linux-image-6.1.16-v8+ (6.1.16-v8+-1) ...
Processing triggers for linux-image-6.1.16-v8+ (6.1.16-v8+-1) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 6.1.16-v8+
   ...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.1.16-v8+
Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
flash-kernel: deferring update (trigger activated)
/etc/kernel/postinst.d/zz-flash-kernel:
Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
flash-kernel: deferring update (trigger activated)
Processing triggers for flash-kernel (3.106ubuntu13) ...
+ [ xtriggered = xtriggered ]
+ FLASH_KERNEL_NOTRIGGER=y flash-kernel
Using DTB: bcm2711-rpi-4-b.dtb
Installing /etc/flash-kernel/dtbs/bcm2711-rpi-4-b.dtb into /boot/dtbs/6.1.16-v8+/./bcm2711-rpi-4-b.dtb
Taking backup of bcm2711-rpi-4-b.dtb.
Installing new bcm2711-rpi-4-b.dtb.
dpkg-query: no packages found matching
+ exit 0

(without the exit 0, flash-kernel always quits with an error, leaving the install of the kernel broken.)

I can't tell if the flash-kernel breakage is connected to this change or not... but I can't understand why there is a dpkg-query error here.

I do see this:
 dpkg-query -l linux-image-6.1.16-v8+
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-======================-============-============-=================================
ii linux-image-6.1.16-v8+ 6.1.16-v8+-1 arm64 Linux kernel, version 6.1.16-v8+

Revision history for this message
satmandu (satadru-umich) wrote :

(Once I've installed the kernel by manually copying the generated initrd.img and vmlinuz into /boot/firmware/ I can boot with the new kernel just fine, but the manual install wasn't needed a month ago.)

Dave Jones (waveform)
summary: - flash-kernel failure when upgrading f-k anad kernel in the same cycle
+ [SRU] flash-kernel failure when upgrading f-k anad kernel in the same
+ cycle
Steve Langasek (vorlon)
summary: - [SRU] flash-kernel failure when upgrading f-k anad kernel in the same
+ [SRU] flash-kernel failure when upgrading f-k and kernel in the same
cycle
Revision history for this message
Steve Langasek (vorlon) wrote :

I am concerned that the way that this is being handled is significantly different from the well-exercised path used on x86. But I can't actually work out from the code how we are guarding against this with grub, despite the fact that I know we had a lot of design discussions about it previously. I'm cc:ing Andy Whitcroft here to see if he recalls what we're doing (if anything!) on x86 as I would like to see cross-architecture alignment here.

Changed in flash-kernel (Ubuntu Kinetic):
status: New → Incomplete
Revision history for this message
Dave Jones (waveform) wrote :

I'll try and explain my understanding of the differences between GRUB-based (x86) systems and flash-kernel systems (ARM, RISCV, x86 using u-boot, etc.), but I admit my knowledge of the GRUB side is considerably less than the flash-kernel side:

On all systems, the kernel packaging installs the kernel (and its device-trees, if they exist) into /boot, and initramfs-tools generates the initrd in /boot.

On GRUB-based systems, /boot may be part of the root partition (when root is not encrypted), or it may be its own partition (when root is encrypted) but in either case it's *typically* an ext-based file-system (ext2, ext4, etc). No further copying of the kernel or initrd is required.

On flash-kernel systems, /boot is always part of the root partition. It's "just another directory". The kernel or initrd then needs copying / flashing to some other location.

On GRUB-based systems, the boot (whether BIOS or EFI based) will eventually reach a GRUB stage which understands how to read the ext-based file-system, from where it'll load the kernel and initrd and proceed from there.

On flash-kernel systems, the bootloader (u-boot, pi-boot, aboot, etc.) is typically configured only to read FAT (some of these bootloaders *can* be configured to read ext-fs but aren't, usually due to performance / stability concerns). Hence, flash-kernel is called to copy artefacts from /boot to wherever they're required (some FAT file-system, or perhaps an EEPROM block).

Hence, on GRUB-based systems, there need not be a guard against the initrd not being generated before GRUB is updated. Even if the GRUB config is generated before the initrd exists, it doesn't matter because it will by the time we next boot (and the config just "points at" the file under /boot). However, on flash-kernel systems, the initrd under /boot is being actively transferred elsewhere and therefore it *does* matter that it exists prior to f-k being called.

At least, that's my understanding!

Revision history for this message
Steve Langasek (vorlon) wrote :

Dave, I don't agree that there's a relevant difference here between flash-kernel and grub.

Grub: initrd not yet generated; grub postinst is called and adds kernel as first entry in grub.cfg; power event happens before kernel postinst is called; system fails to (noninteractively) boot.

flash-kernel: initrd not yet generated; flash-kernel postinst is called and writes kernel (without initramfs) to the boot partition; power event happens before kernel postinst is called; system fails to boot.

Both scenarios need to be guarded against, and we should use the same model for both (and any other bootloaders on other archs).

*Unfortunately*, it turns out that while Andy and I whiteboarded this about 5 years back, what we discussed never wound up being implemented in the kernel packages. I believe the plan at the time was to use a dpkg-divert in the kernel preinst to divert /boot/vmlinuz-$ver to /boot/vmlinuz-$ver.dpkg-something, and then to un-divert it in the postinst only after the initramfs has been generated. grub already has code to ignore the pattern vmlinuz*.dpkg-* so no further changes would be required for grub; and it would be easy to make flash-kernel match.

However, none of this matters so long as the kernel doesn't implement such an interface.

In any case, I'm not happy with the proposed implementation in flash-kernel. I think rooting around in the dpkg database is bad form. Ultimately, though, it's not my opinion on this that matters - we really need input from the Kernel Team here wrt what interface we're going to support between the packages. I subscribed Andy to the bug, but he hasn't replied yet; adding Dimitri as well.

Revision history for this message
Steve Langasek (vorlon) wrote :

Nothing currently awaiting sponsorship, unsubscribing ~ubuntu-sponsors.

Revision history for this message
Dave Jones (waveform) wrote :

On Fri, Jun 23, 2023 at 05:40:57PM -0000, Steve Langasek wrote:
> Dave, I don't agree that there's a relevant difference here between
> flash-kernel and grub.
>
> Grub: initrd not yet generated; grub postinst is called and adds
> kernel as first entry in grub.cfg; power event happens before kernel
> postinst is called; system fails to (noninteractively) boot.
>
> flash-kernel: initrd not yet generated; flash-kernel postinst is
> called and writes kernel (without initramfs) to the boot partition;
> power event happens before kernel postinst is called; system fails to
> boot.
>
> Both scenarios need to be guarded against, and we should use the same
> model for both (and any other bootloaders on other archs).

The "system loses power while installing the kernel" is an extremely
variable story on the flash-kernel side of things. On some platforms
it's mostly guarded against, on others (including the Pi) it's not (and
never has been), and fixing this particular bug will not change that.

> *Unfortunately*, it turns out that while Andy and I whiteboarded this
> about 5 years back, what we discussed never wound up being implemented
> in the kernel packages. I believe the plan at the time was to use a
> dpkg-divert in the kernel preinst to divert /boot/vmlinuz-$ver to
> /boot/vmlinuz-$ver.dpkg-something, and then to un-divert it in the
> postinst only after the initramfs has been generated. grub already
> has code to ignore the pattern vmlinuz*.dpkg-* so no further changes
> would be required for grub; and it would be easy to make flash-kernel
> match.

I see; so "linux-version list" would be modified to ignore the
"vmlinuz*.dpkg-*" pattern, causing flash-kernel to ignore the diverting
kernel as a candidate.

> However, none of this matters so long as the kernel doesn't implement
> such an interface.
>
> In any case, I'm not happy with the proposed implementation in flash-
> kernel. I think rooting around in the dpkg database is bad form.
> Ultimately, though, it's not my opinion on this that matters - we
> really need input from the Kernel Team here wrt what interface we're
> going to support between the packages. I subscribed Andy to the bug,
> but he hasn't replied yet; adding Dimitri as well.

I agree rooting around in dpkg is bad form. I don't *like* this fix, and
I'll be ecstatic if I can chop (most of) it out in future.

So I guess the question is: how long until the kernel folk can implement
the dpkg-divert + linux-version changes, and depending on that answer,
is it worth SRU'ing this in the meantime given the issue has the
potential to break apt upgrades on affected platforms?

Revision history for this message
Robie Basak (racb) wrote :

I very much appreciate Dave's work here, and Steve's input. It seems that we don't yet have consensus on the right way to fix this though? I think Dave's fix is currently in the SRU queue in Kinetic but blocked on consensus then, so setting that bug task to Incomplete.

satmandu's comments 12 and 13 also need addressing please. I'm not sure if using external kernel packages is something we'd support in Ubuntu, and perhaps we wouldn't consider this a regression, but if we're going to knowingly break them in an SRU, we should document that.

Revision history for this message
Robie Basak (racb) wrote :

(oh, it's already Incomplete!)

Revision history for this message
Steve Langasek (vorlon) wrote :

Unfortunately, subscribing Andy and Dimitri hasn't elicited a reply from the Kernel Team on this. And as there are outstanding design/architecture questions, I don't think it's appropriate for us to land this SRU as-is. I'm going to reject the upload from the queue so the SRU team can stop polling it and leave it with you, Dave, to chase the Kernel Team to get consensus here.

Revision history for this message
Steve Langasek (vorlon) wrote : Proposed package upload rejected

An upload of flash-kernel to kinetic-proposed has been rejected from the upload queue for the following reason: "waiting for Kernel Team to weigh in".

Dave Jones (waveform)
tags: added: foundations-todo
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in flash-kernel (Ubuntu Focal):
status: New → Confirmed
Changed in flash-kernel (Ubuntu Jammy):
status: New → Confirmed
Revision history for this message
Michał Fita (michal.fita) wrote :

Recently being his by this on Xilinx ZynqMP while running full `apt upgrade` on vanilla system from the image https://people.canonical.com/~platform/images/xilinx/zcu-ubuntu-22.04/iot-limerick-zcu-classic-desktop-2204-x05-2-20221123-58.img.xz

We have about 20 people affected by this internally with potential to hit more. And this is going to repeat every time the `flash-kernel` is upgraded with the Kernel in the same upgrade flow.

Revision history for this message
Andy Whitcroft (apw) wrote :

With my kernel-team hat on I believe that we are miss-guided fixing this in flash-kernel (or any other consumer of /boot). We are working on a spec to ensure we only have "valid" kernels in /boot.

That said, in the short term though I have no objection to having a flash-kernel specific fix to paper over this.

Revision history for this message
Dave Jones (waveform) wrote :

Setting kinetic to invalid now it's end-of-life. Since the kernel team have now weighed in with no objections to a strictly short-term SRU for the purposes of working around this for existing users I'm going to re-upload the jammy branch and see if we can get this SRU'd.

Changed in flash-kernel (Ubuntu Kinetic):
status: Incomplete → Invalid
Revision history for this message
Steve Langasek (vorlon) wrote :

In the code, you're setting locale variables, e.g.:
LANG=C.UTF-8 dpkg-query --search "$path" 2>/dev/null

I can't work out why this would be needed. What are you trying to change the behavior of here? LC_TYPE? LC_COLLATE? LC_MESSAGES?

In any case, there are several environment variables that take precedence over this setting, so it is probably not actually what you want.

Changed in flash-kernel (Ubuntu Jammy):
status: Confirmed → Incomplete
Revision history for this message
Dave Jones (waveform) wrote :

Scanning the dpkg-query code I think the first instance (dpkg-query --search) is redundant. The history behind it is that the output from dpkg-query for *diverted files* is localized and in the code this originally came from I needed to avoid that localization. However, we're not concerned with diverted files here, and I see the output for non-diverted files doesn't involve localization so there's no need to avoid it.

The second instance (dpkg-query --show) is *possibly* redundant and has to do with whether the db:Status field is localized (or potentially could be) in output. If it's not, then it's redundant. Still, you are correct; I guess LC_ALL would be the better catch-all here (though there's something nagging in my memory about dangers of using LC_ALL)?

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 2007827] Re: [SRU] flash-kernel failure when upgrading f-k and kernel in the same cycle

> though there's something nagging in my memory about dangers of using
> LC_ALL)?

I don't know of any dangers here. It's certainly safer than relying on
$LANG if your script behavior is dependent on the locale. So please fix :)

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

I see that the new version in the queue uses LC_ALL as requested. The rest of the changes seem fine. And I see that the kernel team was +1 on getting this fixed in flash-kernel in the meantime.

Changed in flash-kernel (Ubuntu Jammy):
status: Incomplete → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello ethan.hsieh, or anyone else affected,

Accepted flash-kernel into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/flash-kernel/3.104ubuntu17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
ethan.hsieh (ethan.hsieh) wrote :

Verified the package on jammy-proposed. Can upgrade flash-kernel and linux packages at the same time without any problem.

Reboot the system. The kernel in fitimage is the new version. For detailed log, please see the attached file.
$ uname -a
Linux zynqmp 5.15.0-1025-xilinx-zynqmp #29-Ubuntu SMP Tue Oct 31 01:56:19 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
$ apt-cache policy linux-xilinx-zynqmp flash-kernel | grep Installed
  Installed: 5.15.0.1025.29
  Installed: 3.104ubuntu17

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for flash-kernel has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package flash-kernel - 3.104ubuntu17

---------------
flash-kernel (3.104ubuntu17) jammy; urgency=medium

  * Ensure that only kernels in fully "installed" state are considered for
    flashing (LP: #2007827)
  * d/control: Remove redundant awk dependency to appease lintian
  * d/control: Bump the standards version

 -- Dave Jones <email address hidden> Tue, 06 Jun 2023 11:01:47 +0100

Changed in flash-kernel (Ubuntu Jammy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.