curtin: install flash-kernel in arm64 UEFI unexpected

[Date Huang]

[Impact (linux)]
The only package that currently satisfies the bootloader Recommends relationship on ARM systems is flash-kernel. This ignores EFI-based systems, which will instead require GRUB. Our installers know to install GRUB anyway - but flash-kernel also gets installed. Normally flash-kernel realizes it is not needed and just exits - so the impact is limited to wasting space and CPU cycles on each kernel update. However, there can be cases where calling flash-kernel can cause problems. The original report here describes one where flash-kernel thinks it recognizes the system and tries to run anyway, when in fact GRUB is the correct boot loader.

[Test Case (linux)]
On an arm64 system, confirm that grub-efi-arm64 is an option in the Recommends field:

$ apt show linux-image-unsigned-5.4.0-78-generic | grep Recommends

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Recommends: flash-kernel | grub-efi-arm64, initramfs-tools | linux-initramfs-tool

[What Could Go Wrong (linux)]
First let me describe the mitigations against something going wrong. The proposed patch was already in the hirsute kernel at the time of hirsute GA, so it's had some real world testing, including in our installers. In addition, the patch still leaves flash-kernel as the *default* bootloader Recommends (first in the |'d list) - it only adds grub-efi-arm64 as a secondary option, preventing the installation of flash-kernel if GRUB is already there.

So, the only scenario where I can see a problem might be if something depends on flash-kernel getting installed due to a Recommends even though GRUB is already present.

= Original Report Follows =

I used APM Mustang which flash-kernel supported in u-boot mode.
But I used it with UEFI environment.
It will cause fatal error when I used ARM64 ubuntu live server ISO to install system.

In code[1], this will not install `flash-kernel` for APM Mustang because of UEFI.
So that means code[2] will not disable `flash-kernel` in target system, only disable `update-initramfs`.

When curtin execute to `install_kernel` stage, code[3,4] will not install `flash-kernel` either.
But in code[5], it will install `linux-generic`.
`linux-generic` has a long dependency tree and it will get `flash-kernel` in Recommended field.
Apt by default will install Recommended package before kernel is installed.[6]
So it will still execute `zz-flash-kernel` and `flash-kernel` when installing kernel.
But system didn't create any `initrd.img` ever because curtin disable `update-initramfs` in code[2].
This will cause that `flash-kernel` cannot find `initrd.img.<kvers>` and fail when installing it.

This issue didn't effect all ARM64 UEFI platform because `flash-kernel` didn't support them and skip.[7]
I'm not sure which is best solution for this.
But I think we should apply PR-27 in `flash-kernel`[8] for enhancement and fix curtin process with this patch both.

If we only apply PR-27, it should work fine as well because it will be skipped when detecting UEFI
and install `flash-kernel` before `disable_update_initranfs` in ARM platform without UEFI.[9]

[Patch-1,2,3] might have side effect.
Picking one patch for curtin should be enough.
But I need your advice for this to determine which one is better for curtin.
There are two categories
1. avoid installing flash-kernel if no need, [Patch1,2]
2. always install flash-kernel in arm/arm64 and make sure it be installed before code[2] [Patch3]
(I will attach patch in reply.)

Thanks a lot
Regards,
Date

[1] https://github.com/canonical/curtin/blob/master/curtin/deps/__init__.py#L57-L58
[2] https://github.com/canonical/curtin/blob/master/curtin/commands/curthooks.py#L1693-L1699
[3] https://github.com/canonical/curtin/blob/master/curtin/commands/curthooks.py#L365-L370
[4] https://github.com/canonical/curtin/blob/master/curtin/commands/curthooks.py#L311-L327
[5] https://github.com/canonical/curtin/blob/master/curtin/commands/curthooks.py#L372-L374
[6] https://github.com/Debian/apt/blob/master/apt-pkg/init.cc#L132
[7] https://salsa.debian.org/installer-team/flash-kernel/-/blob/master/functions#L787
[8] https://salsa.debian.org/installer-team/flash-kernel/-/merge_requests/27
[9] curtin will insert `flash-kernel` into `REQUIRED_EXECUTABLES` when system is arm/arm64 without UEFI.

[Date Huang (tjjh89017)]

Patch-1

[Date Huang (tjjh89017)]

Patch-2

[Date Huang (tjjh89017)]

Patch 3

[Date Huang (tjjh89017)]

here is the reason why ARM64 U-boot isn't effected by this issue, and why APM Mustang with UEFI will be effected.
I'm not sure MAAS install this kind of machine will start from cloud-img with existing kernel and initrd.img or not.

Here is the curtin process in installation ISO right now.
1. Determine U-boot/UEFI in arm/arm64 environment.
1.a. If need, install flash-kernel in this stage.
2. Omit something
3. Disable `update-initramfs`.
3.a. This will also disable flash-kernel in /target with ARM64
4. Omit something
5. Install kernel
5.a. this will install kernel with recommended package
5.b. in arm64, this will install flash-kernel

So in arm64 UEFI,
1. Determine U-boot/UEFI in arm/arm64 environment.
1.a. Install flash-kernel because system is under U-boot [correct]
2. Omit something
3. Disable `update-initramfs`.
3.a. Disable flash-kernel because moonshot is under U-boot [correct]
4. Omit something
5. Install kernel
5.a. this will install kernel with recommended package
5.b. in arm64, this will install flash-kernel
5.c. System already installed flash-kernel and disable it in previous stage, so this system will not be effected by this issue. [correct]

In APM Mustang with UEFI,
1. Determine U-boot/UEFI in arm/arm64 environment.
1.a. Will not install flash-kernel because mustang is under UEFI [correct]
2. Omit something
3. Disable `update-initramfs`.
3.a. Will not disable flash-kernel because mustang is under UEFI and flash-kernel isn't installed. [correct]
4. Omit something
5. Install kernel
5.a. this will install kernel with recommended package
5.b. in arm64, this will install flash-kernel [Unexpected]
5.c. System will install flash-kernel. and flash-kernel isn't disabled in previous stage. APM mustang is in flash-kernel supported list. FATAL error here. [Unexpected, fatal]

In Other ARM64 UEFI platform,
1. Determine U-boot/UEFI in arm/arm64 environment.
1.a. Will not install flash-kernel because system is under UEFI [correct]
2. Omit something
3. Disable `update-initramfs`.
3.a. Will not disable flash-kernel because system is under UEFI and flash-kernel isn't installed. [correct]
4. Omit something
5. Install kernel
5.a. this will install kernel with recommended package
5.b. in arm64, this will install flash-kernel [Unexpected]
5.c. System will install flash-kernel. and flash-kernel isn't disabled in previous stage. But flash-kernel will skip because those platforms aren't in supported list. So NO FATAL error here. [Unexpected, no fatal]

Because of these, dpkg-divert doesn't always work as expected.
This bug might occurs again when someone add new platform into flash-kernel and that platform support UEFI and U-Boot both in the future.

By the way, I patch curtin during starting installation ISO with Patch-1, it will never install flash-kernel and install system as expected.

[Date Huang (tjjh89017)]

related to this issue
https://bugs.launchpad.net/subiquity/+bug/1856038

I will submit an issue to Debian `flash-kernel`, and get issue number back to here.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "patch-1" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[dann frazier (dannf)]

I wonder if this could be solved by changing the kernel's

Recommends: flash-kernel

to:

Recommends: grub-efi-arm64 | flash-kernel

That will prevent recommends from installing flash-kernel if grub-efi-arm64 is already installed. This also mirrors the existing Recommends selection for other architectures:

Package: linux-image-unsigned-5.10.0-12-generic
[...]
Recommends: grub-pc [amd64] | grub-efi-amd64 [amd64] | grub-efi-ia32 [amd64] | grub [amd64] | lilo [amd64] | flash-kernel [armhf arm64] | grub-ieee1275 [ppc64el], initramfs-tools | linux-initramfs-tool

Perhaps we can test this by uploading such a kernel to a PPA and doing a test curtin install on an EFI system to confirm that flash-kernel never gets installed. Even if it doesn't fix this issue, still seems like the correct thing to do in the kernel package.

[dann frazier (dannf)]

I uploaded a test build as described in Comment #7 to https://launchpad.net/~dannf/+archive/ubuntu/lp1918427

[Date Huang (tjjh89017)]

Hi Dann

I'm not sure how to test custom kernel with Curtin during installation.
I will try to install it into non-flash-kernel system and see APT will install it or not.

As your advice, I have a question about this.
If this question isn't critical, I think this should work.
Do we need to consider about someone who will want to use EFI-STUB to boot directly?
If so, maybe we shouldn't add grub-efi-arm64 to recommends?
Because it will install unnecessary package for them.

grub-efi-* will be installed in `install_missing_packages` stage before `install_kernel`.
So flash-kernel will not be installed if grub-efi-* is installed and in recommends of kernel.

[dann frazier (dannf)]

I tested this locally by creating a MAAS preseed that adds my PPA. And it works.... sort of. The kernel no longer causes flash-kernel to be installed, but unfortunately flash-kernel is *pre-baked* in the arm* cloud images, so we'd have to remove it from them too.

RE: EFI-STUB - GRUB is the default way to boot Ubuntu on EFI systems. Users can choose to boot using EFI-STUB, but they can also remove any Recommended packages. Adding grub-efi-arm64 to Recommends will also get arm64 on-par with amd64, which also recommends GRUB.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1918427

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[dann frazier (dannf)]

In case it helps, attached is the MAAS preseed I used for testing the PPA kernel w/ the updated Recommends. I also removed flash-kernel from the corresponding MAAS cloud image by locally unsquashfs/purge flash-kernel/mksquashfs and dd'ing back over the existing image.

[dann frazier (dannf)]

Unfortunately dropping flash-kernel from cloud-images causes xgene-uboot deployment failures, see attached log.

[Date Huang (tjjh89017)]

After reading log in #14, and some code from cloud-init and maas
I still cannot find any code that will let cloud-init try to install flash-kernel wtih zfsutils-linux.
and it seems the error because squashfs didn't have any kernel installed.
So `linux-version` cannot find any kernel version to pass to flash-kernel to process postinst script.
`Can't find /boot/vmlinuz- (see /tmp/flash-kernel-no-kernel-error.log)` $kvers is empty.

and as I know, squashfs only for temp utils?
It will create an environment for curtin to install real system into disk
Am I correct?

Do you have any custom cloud-init config for xgene?

> RE: EFI-STUB - GRUB is the default way to boot Ubuntu on EFI systems. Users can choose to boot using EFI-STUB, but they can also remove any Recommended packages. Adding grub-efi-arm64 to Recommends will also get arm64 on-par with amd64, which also recommends GRUB.

Got it.

[dann frazier (dannf)]

On Fri, Mar 12, 2021 at 10:45 PM Date Huang <email address hidden> wrote:
>
> After reading log in #14, and some code from cloud-init and maas
> I still cannot find any code that will let cloud-init try to install flash-kernel wtih zfsutils-linux.
> and it seems the error because squashfs didn't have any kernel installed.
> So `linux-version` cannot find any kernel version to pass to flash-kernel to process postinst script.
> `Can't find /boot/vmlinuz- (see /tmp/flash-kernel-no-kernel-error.log)` $kvers is empty.

I added some debugging and found that this is what causes flash-kernel
to be installed:
  https://git.launchpad.net/curtin/tree/curtin/deps/__init__.py?h=21.2#n57

I'm not sure what the point of that is - it probably should be removed.

> and as I know, squashfs only for temp utils?
> It will create an environment for curtin to install real system into disk
> Am I correct?

Yes, the failure in comment #14 is occurring in what we call the
"ephemeral image", which only exists in memory to install an OS to the
disk. Though the same image is also what is written to the disk.

> Do you have any custom cloud-init config for xgene?

No

  -dann

[dann frazier (dannf)]

Attached is a patch for curtin that works for me, though it could use some cleanup. It installs flash-kernel in the same place GRUB gets installed for EFI-based systems, which seems like the appropriate place. Before installint it, it diverts /usr/sbin/flash-kernel so that flash-kernel's postinst doesn't try (and fail) to flash a kernel that isn't yet installed. There's existing code in enable_update_initramfs() that will undo this diversion. I suspect we should refactor disable_update_initramfs() somehow instead of duplicating this code like I've done here.

I've tested this w/ a locally-hacked squash image that has flash-kernel purged and a PPA kernel that allows grub-efi-arm64 to satisfy the bootloader Recommends. On an EFI-based system, flash-kernel never gets installed. On a Moonshot m400, flash-kernel gets installed as expected.

Once curtin has been updated with a fix like this, and MAAS has been refreshed to include it, I think we should then be able to remove flash-kernel from the arm64 cloud images.

[Date Huang (tjjh89017)]

There is no need to install flash-kernel here.
It will install flash-kernel in "ephemeral image" which didn't have kernel installed.
https://git.launchpad.net/curtin/tree/curtin/deps/__init__.py?h=21.2#n57

So that means, curtin will never install flash-kernel into real system because flash-kernel was already existing via squashfs/cloudimg/or something else.
(curtin will copy files to real system before install kernel and those above we discuss.)

Maybe we can move `install bootloader` after kernel is installed and enable_update_initramfs.
keep `install_missing_packages` as pre
and add `install_missing_pacakges` or `install bootloader_package` as post

[Date Huang (tjjh89017)]

Maybe this WIP patch will work fine and we don't need to refactor disable_update_initramfs.
this patch also consider if there is already initramfs or flash-kernel installed.
Try to guarantee order

But this might override and deactivate bugfix for LP:1640519

[Date Huang (tjjh89017)]

agree with dannf
we should refactor disable_update_initramfs and enable_update_initramfs.
we should not use `which` to determine the executable path.
and always disable update_initramfs or arch-specific package, even those pkgs are not installed.

Is it a good solution that we specific absolute path in disable_update_initramfs?
https://github.com/canonical/curtin/blob/master/curtin/commands/curthooks.py#L117-L124

[Ryan Harper (raharper)]

Hi Dan,

Could you summarize the problem with flash-kernel and this system?

* dann frazier <email address hidden> [2021-03-15 18:25]:
> Attached is a patch for curtin that works for me, though it could use
> some cleanup. It installs flash-kernel in the same place GRUB gets
> installed for EFI-based systems, which seems like the appropriate place.
> Before installint it, it diverts /usr/sbin/flash-kernel so that flash-
> kernel's postinst doesn't try (and fail) to flash a kernel that isn't
> yet installed. There's existing code in enable_update_initramfs() that
> will undo this diversion. I suspect we should refactor
> disable_update_initramfs() somehow instead of duplicating this code like
> I've done here.

Looking at the patch, it looks like you could instead modify
_update_initramfs_tools() to only return flash-kernel in the case that
it's needed rather than for all aarch64.

Alternatively, enable/disable_update_initramfs could take a force
boolean which would perform the operation whether the tool is present
or not (though I don't know if dpkg-divert cares about the presence
of the binary-to-be-diverted).

>
> I've tested this w/ a locally-hacked squash image that has flash-kernel
> purged and a PPA kernel that allows grub-efi-arm64 to satisfy the
> bootloader Recommends. On an EFI-based system, flash-kernel never gets
> installed. On a Moonshot m400, flash-kernel gets installed as expected.
>
> Once curtin has been updated with a fix like this, and MAAS has been
> refreshed to include it, I think we should then be able to remove flash-
> kernel from the arm64 cloud images.

What's the criteria for determining if a particular arm instance
requires flash-kernel or not?

Ryan

[Date Huang (tjjh89017)]

Hi Ryan

> (though I don't know if dpkg-divert cares about the presence
of the binary-to-be-diverted).

I tried dpkg-divert with my mustang and flash-kernel (see in attachment)
It's ok for dpkg-divert even that package isn't installed.

[Date Huang (tjjh89017)]

maybe we can try this patch with alternative path to disable arch-specific binaries before that package is installed.

when `which` didn't get any path for system, we assume that binary isn't installed for now.
and still disable it via alternative path

[Date Huang (tjjh89017)]

attach patch (WIP)

[Ryan Harper (raharper)]

I still don't understand:

1) why does which not find flash-kernel if it's present in the ephemeral image (meaning it will also be present in the target filesystem
2) What is the problem with flash-kernel such that you need to dpkg-divert it?

Generally, we do not want to include paths to binaries; we install Ubuntu and Centos, and possibly other distros which may not have the same path to this tools so I would like to understand what's not working to understand why we're hardcoding paths to binaries for dpkg-divert.

Thanks,
Ryan
Ryan

[dann frazier (dannf)]

On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
>
> Hi Dan,
>
> Could you summarize the problem with flash-kernel and this system?

Sure. flash-kernel recognizes Mustang boards and will generate uImage
and uInitrd files for it, which are required for booting with u-boot
firmware. However, these boards can also run in UEFI mode, which
Date's board does. In UEFI mode, flash-kernel still knows it is on a
Mustang and generates uImage/uInitrd files - which won't be used for
anything in that case, they are just wasting space, but does not cause
it to fail. This does cause problems in a curtin install though.
Curtin has logic to divert away tools that get executed during
initramfs hooks, to avoid failures in packaging scripts before an
initramfs is generated. flash-kernel in particular will fail if an
initramfs is not found on this system. Curtin tries to be smart here
and only divert flash-kernel 1) if it is installed and 2) on systems
that are*not* in UEFI mode, and both of these scenarios have escapes:

1) flash-kernel could get installed post-divert. In that case,
flash-kernel's own postinst will cause it to run and then fail. This
happens today if you start with a cloud image w/o flash-kernel
pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
to be installed along with the kernel. Official cloud images happen to
have flash-kernel pre-baked which avoids this issue. I think curtin
should work whether or not the kernel recommends flash-kernel and
whether or not curtin is pre-baked (in fact, I'd like for us to stop
pre-baking it - the vast majoriy of ARM servers do not need it).

2) If flash-kernel is installed, and curtin finds we're in UEFI mode,
it chooses not to divert flash-kernel. flash-kernel will therefore run
and fail on UEFI Mustangs.

The way I've personally framed this issue is that Ubuntu should not be
trying to install flash-kernel on ARM systems that don't require it,
which is the reason I've added the various tasks here.
 - cloud images shouldn't prebake it
 - the kernel should allow non-flash-kernel bootloaders to satisfy its
recommends
 - curtin shouldn't install flash-kernel on efi-based arm64 servers.
It does this today, but - in what seems like a bug, only in the
ephemeral and not the target.

A separate issue is that flash-kernel should know to just exit if it's
running on an EFI system and not bother creating the unused
uImage/uInitrd - Date recently got a patched merged into Debian's f-k
to do that. That would seemingly also avoid the curtin issues here,
but only if we continue to install flash-kernel all the time.

 -dann

[dann frazier (dannf)]

On Wed, Mar 17, 2021 at 4:56 PM Ryan Harper <email address hidden> wrote:
>
> I still don't understand:
>
> 1) why does which not find flash-kernel if it's present in the ephemeral image (meaning it will also be present in the target filesystem
> 2) What is the problem with flash-kernel such that you need to dpkg-divert it?
>
> Generally, we do not want to include paths to binaries; we install
> Ubuntu and Centos, and possibly other distros which may not have the
> same path to this tools so I would like to understand what's not working
> to understand why we're hardcoding paths to binaries for dpkg-divert.

In my POC patch, it is because we need to divert flash-kernel before
it is installed, *just in case* it gets installed as a dependency of
some other package (in our case it happened due to a kernel Recommends
relationship). We therefore can't use `which` to look up the path...
unless we continue to always install f-k in the ephemeral for the sole
purpose of looking up its path... but that seems to making assumptions
as well - i.e. that the ephemeral OS has the same paths as the target
OS.

[Date Huang (tjjh89017)]

> In my POC patch, it is because we need to divert flash-kernel before
> it is installed, *just in case* it gets installed as a dependency of
> some other package (in our case it happened due to a kernel Recommends
> relationship). We therefore can't use `which` to look up the path...
> unless we continue to always install f-k in the ephemeral for the sole
> purpose of looking up its path... but that seems to making assumptions
> as well - i.e. that the ephemeral OS has the same paths as the target
> OS.

Hi Dann
Maybe we can test my patch in #24

Or we can try to install_kernel with --no-recommends to avoid installing flash-kernel
and move install_missing_packages after install_kernel and enable_update_initramfs?

Thanks
Date

[dann frazier (dannf)]

On Wed, Mar 17, 2021 at 9:10 PM Date Huang <email address hidden> wrote:
>
> > In my POC patch, it is because we need to divert flash-kernel before
> > it is installed, *just in case* it gets installed as a dependency of
> > some other package (in our case it happened due to a kernel Recommends
> > relationship). We therefore can't use `which` to look up the path...
> > unless we continue to always install f-k in the ephemeral for the sole
> > purpose of looking up its path... but that seems to making assumptions
> > as well - i.e. that the ephemeral OS has the same paths as the target
> > OS.
>
> Hi Dann
> Maybe we can test my patch in #24

Date,
  I'm happy to regression test a candidate patch on the xgene/u-boot
and non-xgene/uefi platforms we have once Ryan or some other curtin
maintainer is OK with the approach.

> Or we can try to install_kernel with --no-recommends to avoid installing flash-kernel
> and move install_missing_packages after install_kernel and enable_update_initramfs?

I think the kernel recommending flash-kernel if GRUB is already
installed is just a bug which is now fixed in hirsute:
  https://kernel.ubuntu.com/git/ubuntu/ubuntu-hirsute.git/commit/?id=95325fff0a339a2365a12417a39234e143cc284e

 -dann

[Date Huang (tjjh89017)]

HI Dann

Thanks for your rapid reply!

> I'm happy to regression test a candidate patch on the xgene/u-boot
> and non-xgene/uefi platforms we have once Ryan or some other curtin
> maintainer is OK with the approach.

Sure thanks.

> I think the kernel recommending flash-kernel if GRUB is already
> installed is just a bug which is now fixed in hirsute:
> https://kernel.ubuntu.com/git/ubuntu/ubuntu-hirsute.git/commit/?id=95325fff0a339a2365a12417a39234e143cc284e

Wow, that is fast.
Thanks for this info.

Regards,
Date

[Ryan Harper (raharper)]

* dann frazier <email address hidden> [2021-03-17 20:40]:
> On Wed, Mar 17, 2021 at 4:56 PM Ryan Harper <email address hidden> wrote:
> >
> > I still don't understand:
> >
> > 1) why does which not find flash-kernel if it's present in the ephemeral image (meaning it will also be present in the target filesystem
> > 2) What is the problem with flash-kernel such that you need to dpkg-divert it?
> >
> > Generally, we do not want to include paths to binaries; we install
> > Ubuntu and Centos, and possibly other distros which may not have the
> > same path to this tools so I would like to understand what's not working
> > to understand why we're hardcoding paths to binaries for dpkg-divert.
>
> In my POC patch, it is because we need to divert flash-kernel before
> it is installed, *just in case* it gets installed as a dependency of

Is flash-kernel supposed to be in the cloud-image or not?
Is flash-kernel supposed to be in the target OS or not?

> some other package (in our case it happened due to a kernel Recommends
> relationship). We therefore can't use `which` to look up the path...
> unless we continue to always install f-k in the ephemeral for the sole
> purpose of looking up its path... but that seems to making assumptions
> as well - i.e. that the ephemeral OS has the same paths as the target
> OS.

flash-kernel runs in the target OS; if it's installed in ephemeral
environment thats a by-product of obtaining tools needed but not present
in the ephemeral environment.

curtin/deps.py runs to ensure the ephemeral environment has the needed
tools to create whatever storage configuration curtin supports.

In the squashfs images which do not have a kernel already installed
curtin will install linux-image-generic to make *additional* kernel
modules which may not be in the initrd that maas produces available.
I'm not sure why the maas initrd doesn't yet have zfs kernel modules but
that's typically the driver of the linux-image-generic install.

w.r.t the dep on flash-kernel, if that is only supposed to run in the
target image, then I agree we can remove lines 57-58 in
curtin/dep/__init.py and not install the package (though it may still
get pulled in via the kernel package install).

Later durting curthooks stage, curtin will query the *target* os in
install-missing-packages for expected packages needed in the target OS.

Prior to installing packages in the target OS, curtin does attempt to
divert initramfs triggering tools to prevent generating the initramfs
more than once (this happens when we install a kernel package in the
target OS and either updates or other packages trigger initramfs
rebuilds). This is designed around the assumption that the initramfs
generation tool is present in the image but not tied to the kernel
pacakge. For Ubuntu this is initramfs-tools along with some arch
specific tools. flash-kernel is the tool used on arm for non-uefi boot.

I see a couple of options but I think I'd like to understand:

1) when is flash-kernel required? Curtin current expects flash-kernel
to be needed when arch is arm and when booting non-uefi. Is this still
accurate?

2) Is flash-kernel always installed in arm cloud images us...

[Ryan Harper (raharper)]

* dann frazier <email address hidden> [2021-03-17 20:30]:
> On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> >
> > Hi Dan,
> >
> > Could you summarize the problem with flash-kernel and this system?
>
> Sure. flash-kernel recognizes Mustang boards and will generate uImage
> and uInitrd files for it, which are required for booting with u-boot
> firmware. However, these boards can also run in UEFI mode, which
> Date's board does. In UEFI mode, flash-kernel still knows it is on a
> Mustang and generates uImage/uInitrd files - which won't be used for
> anything in that case, they are just wasting space, but does not cause
> it to fail. This does cause problems in a curtin install though.
> Curtin has logic to divert away tools that get executed during
> initramfs hooks, to avoid failures in packaging scripts before an
> initramfs is generated. flash-kernel in particular will fail if an
> initramfs is not found on this system. Curtin tries to be smart here
> and only divert flash-kernel 1) if it is installed and 2) on systems
> that are*not* in UEFI mode, and both of these scenarios have escapes:
>
> 1) flash-kernel could get installed post-divert. In that case,
> flash-kernel's own postinst will cause it to run and then fail. This
> happens today if you start with a cloud image w/o flash-kernel
> pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> to be installed along with the kernel. Official cloud images happen to

Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
chroot into it and install the linux-image-generic package pulling in
flash-kernel this fails due to postinst of flash-kernel expecting
initramfs to already be generated? This doesn't seem like a curtin bug.

> have flash-kernel pre-baked which avoids this issue. I think curtin
> should work whether or not the kernel recommends flash-kernel and
> whether or not curtin is pre-baked (in fact, I'd like for us to stop
> pre-baking it - the vast majoriy of ARM servers do not need it).

>
> 2) If flash-kernel is installed, and curtin finds we're in UEFI mode,
> it chooses not to divert flash-kernel. flash-kernel will therefore run
> and fail on UEFI Mustangs.

I don't think that's true. The logic for disabling initramfs tools
always runs regardless of UEFI mode and arch. See
curtin/commands/curthooks.py:builtin_curthooks() lines 1692- 1699

>
> The way I've personally framed this issue is that Ubuntu should not be
> trying to install flash-kernel on ARM systems that don't require it,
> which is the reason I've added the various tasks here.
> - cloud images shouldn't prebake it

OK

> - the kernel should allow non-flash-kernel bootloaders to satisfy its
> recommends

OK

Thanks. I replied to your later post without seeing this first.
This helps a lot.

> - curtin shouldn't install flash-kernel on efi-based arm64 servers.
> It does this today, but - in what seems like a bug, only in the
> ephemeral and not the target.

Yeah; I suspect flash-kernel being pre-baked into images hid this for
some time. As I mentioned in the other reply, I do think that lines
57-58 in curtin/deps/__init__.py which bring in fla...

[dann frazier (dannf)]

On Thu, Mar 18, 2021 at 10:11 AM Ryan Harper <email address hidden> wrote:
>
> * dann frazier <email address hidden> [2021-03-17 20:40]:
> > On Wed, Mar 17, 2021 at 4:56 PM Ryan Harper <email address hidden> wrote:
> > >
> > > I still don't understand:
> > >
> > > 1) why does which not find flash-kernel if it's present in the ephemeral image (meaning it will also be present in the target filesystem
> > > 2) What is the problem with flash-kernel such that you need to dpkg-divert it?
> > >
> > > Generally, we do not want to include paths to binaries; we install
> > > Ubuntu and Centos, and possibly other distros which may not have the
> > > same path to this tools so I would like to understand what's not working
> > > to understand why we're hardcoding paths to binaries for dpkg-divert.
> >
> > In my POC patch, it is because we need to divert flash-kernel before
> > it is installed, *just in case* it gets installed as a dependency of
>
> Is flash-kernel supposed to be in the cloud-image or not?

I see no reason for it to be, and I would like to see it removed. For
ARM virtual machines (which I consider, possibly incorrectly, the
primary consumer of cloud images), the only standard way to boot the
kernel-on-disk is booting in EFI mode w/ GRUB.

> Is flash-kernel supposed to be in the target OS or not?

GRUB should be installed on EFI systems. flash-kernel should be
installed on non-EFI systems.
It should be safe to install them both in either case, but one of them
is going to be useless.

> > some other package (in our case it happened due to a kernel Recommends
> > relationship). We therefore can't use `which` to look up the path...
> > unless we continue to always install f-k in the ephemeral for the sole
> > purpose of looking up its path... but that seems to making assumptions
> > as well - i.e. that the ephemeral OS has the same paths as the target
> > OS.
>
> flash-kernel runs in the target OS; if it's installed in ephemeral
> environment thats a by-product of obtaining tools needed but not present
> in the ephemeral environment.
>
> curtin/deps.py runs to ensure the ephemeral environment has the needed
> tools to create whatever storage configuration curtin supports.
>
> In the squashfs images which do not have a kernel already installed
> curtin will install linux-image-generic to make *additional* kernel
> modules which may not be in the initrd that maas produces available.
> I'm not sure why the maas initrd doesn't yet have zfs kernel modules but
> that's typically the driver of the linux-image-generic install.
>
> w.r.t the dep on flash-kernel, if that is only supposed to run in the
> target image, then I agree we can remove lines 57-58 in
> curtin/dep/__init.py and not install the package (though it may still
> get pulled in via the kernel package install).

OK, agreed. Maybe we start by just ripping that out.

>
> Later durting curthooks stage, curtin will query the *target* os in
> install-missing-packages for expected packages needed in the target OS.
>
> Prior to installing packages in the target OS, curtin does attempt to
> divert initramfs triggering tools to prevent generating the initramfs
> more than once ...

[dann frazier (dannf)]

On Thu, Mar 18, 2021 at 10:25 AM Ryan Harper <email address hidden> wrote:
>
> * dann frazier <email address hidden> [2021-03-17 20:30]:
> > On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> > >
> > > Hi Dan,
> > >
> > > Could you summarize the problem with flash-kernel and this system?
> >
> > Sure. flash-kernel recognizes Mustang boards and will generate uImage
> > and uInitrd files for it, which are required for booting with u-boot
> > firmware. However, these boards can also run in UEFI mode, which
> > Date's board does. In UEFI mode, flash-kernel still knows it is on a
> > Mustang and generates uImage/uInitrd files - which won't be used for
> > anything in that case, they are just wasting space, but does not cause
> > it to fail. This does cause problems in a curtin install though.
> > Curtin has logic to divert away tools that get executed during
> > initramfs hooks, to avoid failures in packaging scripts before an
> > initramfs is generated. flash-kernel in particular will fail if an
> > initramfs is not found on this system. Curtin tries to be smart here
> > and only divert flash-kernel 1) if it is installed and 2) on systems
> > that are*not* in UEFI mode, and both of these scenarios have escapes:
> >
> > 1) flash-kernel could get installed post-divert. In that case,
> > flash-kernel's own postinst will cause it to run and then fail. This
> > happens today if you start with a cloud image w/o flash-kernel
> > pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> > to be installed along with the kernel. Official cloud images happen to
>
> Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
> chroot into it and install the linux-image-generic package pulling in
> flash-kernel this fails due to postinst of flash-kernel expecting
> initramfs to already be generated? This doesn't seem like a curtin bug.

If done so in a chroot that exposes the kernel interfaces (/proc &
/sys) that claim to be hardware that requires the initramfs to be
post-processed, yes.
I suppose f-k could be taught to detect it is in a chroot, but then
again - depending on what you are building the chroot for - you may
want that post-processing.

> > have flash-kernel pre-baked which avoids this issue. I think curtin
> > should work whether or not the kernel recommends flash-kernel and
> > whether or not curtin is pre-baked (in fact, I'd like for us to stop
> > pre-baking it - the vast majoriy of ARM servers do not need it).
>
> >
> > 2) If flash-kernel is installed, and curtin finds we're in UEFI mode,
> > it chooses not to divert flash-kernel. flash-kernel will therefore run
> > and fail on UEFI Mustangs.
>
> I don't think that's true. The logic for disabling initramfs tools
> always runs regardless of UEFI mode and arch. See
> curtin/commands/curthooks.py:builtin_curthooks() lines 1692- 1699

oh, you're right. I mistakenly thought it was calling
get_flash_kernel_pkgs(), which returns nothing if EFI is detected.

> >
> > The way I've personally framed this issue is that Ubuntu should not be
> > trying to install flash-kernel on ARM systems that don't require it,
> > which is the reason I'...

[Ryan Harper (raharper)]

* dann frazier <email address hidden> [2021-03-18 12:11]:
> On Thu, Mar 18, 2021 at 10:25 AM Ryan Harper <email address hidden> wrote:
> >
> > * dann frazier <email address hidden> [2021-03-17 20:30]:
> > > On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> > > >
> > > > Hi Dan,
> > > >
> > >
> > > 1) flash-kernel could get installed post-divert. In that case,
> > > flash-kernel's own postinst will cause it to run and then fail. This
> > > happens today if you start with a cloud image w/o flash-kernel
> > > pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> > > to be installed along with the kernel. Official cloud images happen to
> >
> > Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
> > chroot into it and install the linux-image-generic package pulling in
> > flash-kernel this fails due to postinst of flash-kernel expecting
> > initramfs to already be generated? This doesn't seem like a curtin bug.
>
> If done so in a chroot that exposes the kernel interfaces (/proc &
> /sys) that claim to be hardware that requires the initramfs to be
> post-processed, yes.

Maybe I'm missing something but if I install linux-image-generic
it populates /boot with vmlinuz-$version (and a few more things)
and /lib/modules/$version and the kernels postinst will invoke
update-initramfs. The /boot/initrd.img-$version is *generated* at
that time during the kernel's postinstall

Now, in the arm case IIUC, the kernel package has a dep on flash-kernel
being present as it's "needed" to generate the initramfs ... so how can
flash-kernel's postinst *fail* if it is the tool that's generating said
initramfs file?

> > In summary curtin will need:
> >
> > move ephemeral deps.py flash-kernel to arch-packages in
> > install-missing-packages with the same logic guarding when to add the
> > dep.
> >
> > It's not clear to me why curtin should work around the packaging bugs
> > around flash-kernel and I would suggest that flash-kernel be kept in the
> > cloud images until the packging deps/bugs around it are fixed.
>
> I don't think it should - we should SRU Date's f-k change and the
> kernel Recommends change. Are there other packaging issues you've
> identified?

I don't think possibly something flash-kernel related to the above
discussion but that may just be my own misinformation.

[dann frazier (dannf)]

On Thu, Mar 18, 2021 at 12:25 PM Ryan Harper <email address hidden> wrote:
>
> * dann frazier <email address hidden> [2021-03-18 12:11]:
> > On Thu, Mar 18, 2021 at 10:25 AM Ryan Harper <email address hidden> wrote:
> > >
> > > * dann frazier <email address hidden> [2021-03-17 20:30]:
> > > > On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> > > > >
> > > > > Hi Dan,
> > > > >
> > > >
> > > > 1) flash-kernel could get installed post-divert. In that case,
> > > > flash-kernel's own postinst will cause it to run and then fail. This
> > > > happens today if you start with a cloud image w/o flash-kernel
> > > > pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> > > > to be installed along with the kernel. Official cloud images happen to
> > >
> > > Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
> > > chroot into it and install the linux-image-generic package pulling in
> > > flash-kernel this fails due to postinst of flash-kernel expecting
> > > initramfs to already be generated? This doesn't seem like a curtin bug.
> >
> > If done so in a chroot that exposes the kernel interfaces (/proc &
> > /sys) that claim to be hardware that requires the initramfs to be
> > post-processed, yes.
>
> Maybe I'm missing something but if I install linux-image-generic
> it populates /boot with vmlinuz-$version (and a few more things)
> and /lib/modules/$version and the kernels postinst will invoke
> update-initramfs. The /boot/initrd.img-$version is *generated* at
> that time during the kernel's postinstall
>
> Now, in the arm case IIUC, the kernel package has a dep on flash-kernel
> being present as it's "needed" to generate the initramfs ... so how can
> flash-kernel's postinst *fail* if it is the tool that's generating said
> initramfs file?

What flash-kernel does is generate wrapped versions of *exisiting*
vmlinuz and initrd.img files. It doesn't generate those files, rather
post-processes them.
The kernel doesn't depend on flash-kernel, it just recommends it like
it does GRUB on x86.

  -dann

[Ryan Harper (raharper)]

* dann frazier <email address hidden> [2021-03-18 16:30]:
> On Thu, Mar 18, 2021 at 12:25 PM Ryan Harper <email address hidden> wrote:
> >
> > * dann frazier <email address hidden> [2021-03-18 12:11]:
> > > On Thu, Mar 18, 2021 at 10:25 AM Ryan Harper <email address hidden> wrote:
> > > >
> > > > * dann frazier <email address hidden> [2021-03-17 20:30]:
> > > > > On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> > > > > >
> > > > > > Hi Dan,
> > > > > >
> > > > >
> > > > > 1) flash-kernel could get installed post-divert. In that case,
> > > > > flash-kernel's own postinst will cause it to run and then fail. This
> > > > > happens today if you start with a cloud image w/o flash-kernel
> > > > > pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> > > > > to be installed along with the kernel. Official cloud images happen to
> > > >
> > > > Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
> > > > chroot into it and install the linux-image-generic package pulling in
> > > > flash-kernel this fails due to postinst of flash-kernel expecting
> > > > initramfs to already be generated? This doesn't seem like a curtin bug.
> > >
> > > If done so in a chroot that exposes the kernel interfaces (/proc &
> > > /sys) that claim to be hardware that requires the initramfs to be
> > > post-processed, yes.
> >
> > Maybe I'm missing something but if I install linux-image-generic
> > it populates /boot with vmlinuz-$version (and a few more things)
> > and /lib/modules/$version and the kernels postinst will invoke
> > update-initramfs. The /boot/initrd.img-$version is *generated* at
> > that time during the kernel's postinstall
> >
> > Now, in the arm case IIUC, the kernel package has a dep on flash-kernel
> > being present as it's "needed" to generate the initramfs ... so how can
> > flash-kernel's postinst *fail* if it is the tool that's generating said
> > initramfs file?
>
> What flash-kernel does is generate wrapped versions of *exisiting*
> vmlinuz and initrd.img files. It doesn't generate those files, rather
> post-processes them.
> The kernel doesn't depend on flash-kernel, it just recommends it like
> it does GRUB on x86.

Yes, I get that but it still looks like a packaging bug if dpkg installs
flash-kernel first and /boot is not populated with existing initrds; one
could easily see this happen in a debootstrap.

Is the "liveness" of the chroot what's tripping up flash-kernel? We
currently run inside a chroot which mounts /dev /proc /run and /sys; we
could drop those but it also seems reasonable to have flash-kernel not
expect existing initrds?

[dann frazier (dannf)]

On Fri, Mar 19, 2021 at 10:01 AM Ryan Harper <email address hidden> wrote:
>
> * dann frazier <email address hidden> [2021-03-18 16:30]:
> > On Thu, Mar 18, 2021 at 12:25 PM Ryan Harper <email address hidden> wrote:
> > >
> > > * dann frazier <email address hidden> [2021-03-18 12:11]:
> > > > On Thu, Mar 18, 2021 at 10:25 AM Ryan Harper <email address hidden> wrote:
> > > > >
> > > > > * dann frazier <email address hidden> [2021-03-17 20:30]:
> > > > > > On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> > > > > > >
> > > > > > > Hi Dan,
> > > > > > >
> > > > > >
> > > > > > 1) flash-kernel could get installed post-divert. In that case,
> > > > > > flash-kernel's own postinst will cause it to run and then fail. This
> > > > > > happens today if you start with a cloud image w/o flash-kernel
> > > > > > pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> > > > > > to be installed along with the kernel. Official cloud images happen to
> > > > >
> > > > > Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
> > > > > chroot into it and install the linux-image-generic package pulling in
> > > > > flash-kernel this fails due to postinst of flash-kernel expecting
> > > > > initramfs to already be generated? This doesn't seem like a curtin bug.
> > > >
> > > > If done so in a chroot that exposes the kernel interfaces (/proc &
> > > > /sys) that claim to be hardware that requires the initramfs to be
> > > > post-processed, yes.
> > >
> > > Maybe I'm missing something but if I install linux-image-generic
> > > it populates /boot with vmlinuz-$version (and a few more things)
> > > and /lib/modules/$version and the kernels postinst will invoke
> > > update-initramfs. The /boot/initrd.img-$version is *generated* at
> > > that time during the kernel's postinstall
> > >
> > > Now, in the arm case IIUC, the kernel package has a dep on flash-kernel
> > > being present as it's "needed" to generate the initramfs ... so how can
> > > flash-kernel's postinst *fail* if it is the tool that's generating said
> > > initramfs file?
> >
> > What flash-kernel does is generate wrapped versions of *exisiting*
> > vmlinuz and initrd.img files. It doesn't generate those files, rather
> > post-processes them.
> > The kernel doesn't depend on flash-kernel, it just recommends it like
> > it does GRUB on x86.
>
> Yes, I get that but it still looks like a packaging bug if dpkg installs
> flash-kernel first and /boot is not populated with existing initrds; one
> could easily see this happen in a debootstrap.

Given that a failure to produce a wrapped initrd could cause a system
to become unbootable, it does seem to me like a hard failure here is
warranted. But, perhaps we could provide a "shhhh... it's ok, just
chill" mechanism. Maybe a FLASH_KERNEL_SKIP=1 environment variable?

> Is the "liveness" of the chroot what's tripping up flash-kernel? We
> currently run inside a chroot which mounts /dev /proc /run and /sys; we
> could drop those but it also seems reasonable to have flash-kernel not
> expect existing initrds?

Certainly a non-live chroot can avoid this by leading f-...

[Ryan Harper (raharper)]

* dann frazier <email address hidden> [2021-03-19 12:16]:
> On Fri, Mar 19, 2021 at 10:01 AM Ryan Harper <email address hidden> wrote:
> >
> > * dann frazier <email address hidden> [2021-03-18 16:30]:
> > > On Thu, Mar 18, 2021 at 12:25 PM Ryan Harper <email address hidden> wrote:
> > > >
> > > > * dann frazier <email address hidden> [2021-03-18 12:11]:
> > > > > On Thu, Mar 18, 2021 at 10:25 AM Ryan Harper <email address hidden> wrote:
> > > > > >
> > > > > > * dann frazier <email address hidden> [2021-03-17 20:30]:
> > > > > > > On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> > > > > > > >
> > > > > > > > Hi Dan,
> > > > > > > >
> > > > > > >
> > > > > > > 1) flash-kernel could get installed post-divert. In that case,
> > > > > > > flash-kernel's own postinst will cause it to run and then fail. This
> > > > > > > happens today if you start with a cloud image w/o flash-kernel
> > > > > > > pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> > > > > > > to be installed along with the kernel. Official cloud images happen to
> > > > > >
> > > > > > Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
> > > > > > chroot into it and install the linux-image-generic package pulling in
> > > > > > flash-kernel this fails due to postinst of flash-kernel expecting
> > > > > > initramfs to already be generated? This doesn't seem like a curtin bug.
> > > > >
> > > > > If done so in a chroot that exposes the kernel interfaces (/proc &
> > > > > /sys) that claim to be hardware that requires the initramfs to be
> > > > > post-processed, yes.
> > > >
> > > > Maybe I'm missing something but if I install linux-image-generic
> > > > it populates /boot with vmlinuz-$version (and a few more things)
> > > > and /lib/modules/$version and the kernels postinst will invoke
> > > > update-initramfs. The /boot/initrd.img-$version is *generated* at
> > > > that time during the kernel's postinstall
> > > >
> > > > Now, in the arm case IIUC, the kernel package has a dep on flash-kernel
> > > > being present as it's "needed" to generate the initramfs ... so how can
> > > > flash-kernel's postinst *fail* if it is the tool that's generating said
> > > > initramfs file?
> > >
> > > What flash-kernel does is generate wrapped versions of *exisiting*
> > > vmlinuz and initrd.img files. It doesn't generate those files, rather
> > > post-processes them.
> > > The kernel doesn't depend on flash-kernel, it just recommends it like
> > > it does GRUB on x86.
> >
> > Yes, I get that but it still looks like a packaging bug if dpkg installs
> > flash-kernel first and /boot is not populated with existing initrds; one
> > could easily see this happen in a debootstrap.
>
> Given that a failure to produce a wrapped initrd could cause a system
> to become unbootable, it does seem to me like a hard failure here is
> warranted. But, perhaps we could provide a "shhhh... it's ok, just
> chill" mechanism. Maybe a FLASH_KERNEL_SKIP=1 environment variable?

Agreed but with the condition that the *input* is present. If the
initrd file has not yet been generated then another error from
...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.11.0-13.14

---------------
linux (5.11.0-13.14) hirsute; urgency=medium

  * CVE-2020-27170
    - bpf: Prohibit alu ops for pointer types not defining ptr_limit
    - bpf, selftests: Fix up some test_verifier cases for unprivileged

  * CVE-2020-27171
    - bpf: Fix off-by-one for area size in creating mask to left

 -- Andrea Righi <email address hidden> Fri, 19 Mar 2021 16:49:32 +0100

[dann frazier (dannf)]

On Fri, Mar 19, 2021 at 1:31 PM Ryan Harper <email address hidden> wrote:
>
> * dann frazier <email address hidden> [2021-03-19 12:16]:
> > On Fri, Mar 19, 2021 at 10:01 AM Ryan Harper <email address hidden> wrote:
> > >
> > > * dann frazier <email address hidden> [2021-03-18 16:30]:
> > > > On Thu, Mar 18, 2021 at 12:25 PM Ryan Harper <email address hidden> wrote:
> > > > >
> > > > > * dann frazier <email address hidden> [2021-03-18 12:11]:
> > > > > > On Thu, Mar 18, 2021 at 10:25 AM Ryan Harper <email address hidden> wrote:
> > > > > > >
> > > > > > > * dann frazier <email address hidden> [2021-03-17 20:30]:
> > > > > > > > On Tue, Mar 16, 2021 at 10:05 AM Ryan Harper <email address hidden> wrote:
> > > > > > > > >
> > > > > > > > > Hi Dan,
> > > > > > > > >
> > > > > > > >
> > > > > > > > 1) flash-kernel could get installed post-divert. In that case,
> > > > > > > > flash-kernel's own postinst will cause it to run and then fail. This
> > > > > > > > happens today if you start with a cloud image w/o flash-kernel
> > > > > > > > pre-baked because Ubuntu's kernel recommends flash-kernel, causing it
> > > > > > > > to be installed along with the kernel. Official cloud images happen to
> > > > > > >
> > > > > > > Hrm, so if we take a squashfs rootfs (with no flash-kernel present)
> > > > > > > chroot into it and install the linux-image-generic package pulling in
> > > > > > > flash-kernel this fails due to postinst of flash-kernel expecting
> > > > > > > initramfs to already be generated? This doesn't seem like a curtin bug.
> > > > > >
> > > > > > If done so in a chroot that exposes the kernel interfaces (/proc &
> > > > > > /sys) that claim to be hardware that requires the initramfs to be
> > > > > > post-processed, yes.
> > > > >
> > > > > Maybe I'm missing something but if I install linux-image-generic
> > > > > it populates /boot with vmlinuz-$version (and a few more things)
> > > > > and /lib/modules/$version and the kernels postinst will invoke
> > > > > update-initramfs. The /boot/initrd.img-$version is *generated* at
> > > > > that time during the kernel's postinstall
> > > > >
> > > > > Now, in the arm case IIUC, the kernel package has a dep on flash-kernel
> > > > > being present as it's "needed" to generate the initramfs ... so how can
> > > > > flash-kernel's postinst *fail* if it is the tool that's generating said
> > > > > initramfs file?
> > > >
> > > > What flash-kernel does is generate wrapped versions of *exisiting*
> > > > vmlinuz and initrd.img files. It doesn't generate those files, rather
> > > > post-processes them.
> > > > The kernel doesn't depend on flash-kernel, it just recommends it like
> > > > it does GRUB on x86.
> > >
> > > Yes, I get that but it still looks like a packaging bug if dpkg installs
> > > flash-kernel first and /boot is not populated with existing initrds; one
> > > could easily see this happen in a debootstrap.
> >
> > Given that a failure to produce a wrapped initrd could cause a system
> > to become unbootable, it does seem to me like a hard failure here is
> > warranted. But, perhaps we could provide a "shhhh... it's ok, just
> > chill" m...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[dann frazier (dannf)]

= focal kernel verification =
ubuntu@aurorus:~$ apt show linux-image-5.4.0-81-generic | grep Recommends

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Recommends: flash-kernel | grub-efi-arm64, initramfs-tools | linux-initramfs-tool
ubuntu@aurorus:~$ apt show linux-image-unsigned-5.4.0-81-generic | grep Recommends

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Recommends: flash-kernel | grub-efi-arm64, initramfs-tools | linux-initramfs-tool
ubuntu@aurorus:~$

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-81.91

---------------
linux (5.4.0-81.91) focal; urgency=medium

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  * large_dir in ext4 broken (LP: #1933074)
    - SAUCE: ext4: fix directory index node split corruption

  * Some test in kselftest/net on focal source tree were not tested at all
    (LP: #1934282)
    - selftests/net: add missing tests to Makefile

  * curtin: install flash-kernel in arm64 UEFI unexpected (LP: #1918427)
    - [Packaging] Allow grub-efi-arm* to satisfy recommends on ARM

  * Add l2tp.sh in net from ubuntu_kernel_selftests back (LP: #1934293)
    - Revert "UBUNTU: SAUCE: selftests/net -- disable l2tp.sh test"

  * icmp_redirect.sh in net from ubuntu_kernel_selftests failed on F-OEM-5.6 /
    F-OEM-5.10 / F-OEM-5.13 / F / G / H (LP: #1880645)
    - selftests: icmp_redirect: support expected failures

  * Focal update: v5.4.128 upstream stable release (LP: #1934179)
    - dmaengine: ALTERA_MSGDMA depends on HAS_IOMEM
    - dmaengine: QCOM_HIDMA_MGMT depends on HAS_IOMEM
    - dmaengine: stedma40: add missing iounmap() on error in d40_probe()
    - afs: Fix an IS_ERR() vs NULL check
    - mm/memory-failure: make sure wait for page writeback in memory_failure
    - kvm: LAPIC: Restore guard to prevent illegal APIC register access
    - batman-adv: Avoid WARN_ON timing related checks
    - net: ipv4: fix memory leak in netlbl_cipsov4_add_std
    - vrf: fix maximum MTU
    - net: rds: fix memory leak in rds_recvmsg
    - net: lantiq: disable interrupt before sheduling NAPI
    - udp: fix race between close() and udp_abort()
    - rtnetlink: Fix regression in bridge VLAN configuration
    - net/sched: act_ct: handle DNAT tuple collision
    - net/mlx5e: Remove dependency in IPsec initialization flows
    - net/mlx5e: Fix page reclaim for dead peer hairpin
    - net/mlx5: Consider RoCE cap before init RDMA resources
    - net/mlx5e: allow TSO on VXLAN over VLAN topologies
    - net/mlx5e: Block offload of outer header csum for UDP tunnels
    - netfilter: synproxy: Fix out of bounds when parsing TCP options
    - sch_cake: Fix out of bounds when parsing TCP options and header
    - alx: Fix an error handling path in 'alx_probe()'
    - net: stmmac: dwmac1000: Fix extended MAC address registers definition
    - net: make get_net_ns return error if NET_NS is disabled
    - qlcnic: Fix an error handling path in 'qlcnic_probe()'
    - netxen_nic: Fix an error handling path in 'netxen_nic_probe()'
    - net: qrtr: fix OOB Read in qrtr_endpoint_post
    - ptp: improve max_adj check against unreasonable values
    - net: cdc_ncm: switch to eth%d interface naming
    - lantiq: net: fix duplicated skb in rx descriptor ring
    - net: usb: fix possible use-after-free in smsc75xx_bind
    - net: fec_ptp: fix issue caused by refactor the fec_devtype
    - net: ipv4: fix memory leak in ip_mc_add1_src
    - net/af_unix: fix a data-race in unix_dgram_sendmsg / unix_release_sock
    - be2net: Fix an error handling path in 'be_probe()'
    - net: hamradio: fix memory leak in mkiss_close
    - net: cdc_eem: fix tx fixup skb leak
    - cxgb4: fix wrong shift.
    - bnx...

[Brian Murray (brian-murray)]

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

[dann frazier (dannf)]

Any objection to dropping flash-kernel from ARM cloud images starting w/ jammy+1?