Can not use newer kernel than comes with the ISO

[Brian Peschel]

Trying to create an Ubuntu live server custom image. The ISO I have downloaded (ubuntu-18.04-live-server-amd64.iso) is using kernel 4.15.0-20.21. As part of the chroot process, I am doing an apt update and apt upgrade. This provides a new kernel, 4.15.0.23.25. If I select the 23 kernel after the chroot page, the resulting kernel image throws a kernel panic when booting.

Interestingly, if I select the 20 kernel and do the install, the new kernel is running on the resulting system:
webadmin@hl-test:~$ dpkg --list | grep linux-headers
ii linux-headers-4.15.0-20 4.15.0-20.21 all Header files related to Linux kernel version 4.15.0
ii linux-headers-4.15.0-20-generic 4.15.0-20.21 amd64 Linux kernel headers for version 4.15.0 on 64 bit x86 SMP
ii linux-headers-4.15.0-23 4.15.0-23.25 all Header files related to Linux kernel version 4.15.0
ii linux-headers-4.15.0-23-generic 4.15.0-23.25 amd64 Linux kernel headers for version 4.15.0 on 64 bit x86 SMP
ii linux-headers-generic 4.15.0.23.25 amd64 Generic Linux kernel headers
webadmin@hl-test:~$ uname -r
4.15.0-23-generic

[Cubic PPA (cubic-wizard)]

This only seems to happen with the Server version.
I get the same result installing other kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/.

[Ray Sherwin (dark-sky)]

I have also tried to use a different kernel than what is in the original iso using ubuntu 18.04 and linux mint 19 images. My new kernel is compiled using "make deb-pkg" and installed in cubic root using dpkg -i linux*.deb. This kernel works just fine on my linux mint 19, ubuntu 18 and Arch Linux installs. The packages installs just fine except a warning that can not determine root from fstab file. When I click next after making all of my changes in the chroot it exits chroot enviorment and nothing happens after pressing next button. If I click the back button and choose Create a disk image button and click the next button cubic just hangs and does nothing.

Get vmlinuz version from file type..... /home/ubuntu/image/squashfs-
                                            root/boot/vmlinuz-4.18.0-rc2-udl+
    Execute synchronously.................. file "/home/ubuntu/image/squashfs-
                                            root/boot/vmlinuz-4.18.0-rc2-udl+"
    No processes were found that are
    using directory........................ /home/ubuntu/image/squashfs-root
    Get mount points in directory.......... /home/ubuntu/image/squashfs-root
    Execute synchronously.................. mount
    Get vmlinuz version from file
    contents............................... /home/ubuntu/image/squashfs-
                                            root/boot/vmlinuz-4.18.0-rc2-udl+
    Set a new process for thread id........ 140620711057152
    The new process id is.................. 10504
    No mount points were found in.......... /home/ubuntu/image/squashfs-root
    Finished running thread with id........ 140620711057152
    Exception encountered in thread........ 140620413097728
    Ignore exception?...................... No
    The tracekback is...................... Traceback (most recent call last):
                                            File
                                            "/usr/share/cubic/transition.py",
                                            line 95, in run self.action()
                                            File
                                            "/usr/share/cubic/transition.py",
                                            line 400, in action self) File
                                            "/usr/share/cubic/transitions.py",
                                            line 2564, in transition__from__term
                                            inal_page__to__manage_options_page
                                            directory_2) File
                                            "/usr/share/cubic/utilities.py",
                                            line 2072, in
                                            create_vmlinuz_version_details_list
                                            get_vmlinuz_versions_list_from_direc
                          ...

[Ray Sherwin (dark-sky)]

I forgot to say the warning I mentioned above "can not determine root from fstab file" is when the dpkg install is creating the initrd.img-4.18.0-rc2-udl+. I have noticed that the original iso image does not much of an initrd except microcode for an intel. Should I create a kernel so an initrd is not required; but that does not make much sense since a kernel package from the normal repo would create an initrd.img. I even created soft links in boot/ for initrd.img and vmlinuz pointing to initrd.img-4.18.0-rc2-udl+ and vmlinuz-4.18.0-rc2-udl+ respectively.

[Cubic PPA (cubic-wizard)]

This only happens with the server image, and I still don't have a resolution.

[William Stearns (wstearns)]

This is happening as well with the 18.04.1 builds of desktop.

[Cubic PPA (cubic-wizard)]

William,

This does ~not~ happen in 18.04 desktop, but it ~does~ happen in 18.04.1 desktop?

[Ray Sherwin (dark-sky)]

After compiling the kernel with the aufs patches applied I can boot just fine with UEFI but not legacy for some reason. At least there is progress now. I am tired of messing with it for today and will start back with it tomorrow when I am fresh to see where I messed up creating the .iso involving the legacy boot.

[Cubic PPA (cubic-wizard)]

Ray,

Look at line 3012 in utilities.py to see what is being executed.
The function create_iso_image(thread) uses xorriso or mkisofs to create the iso...

https://bazaar.launchpad.net/~cubic-wizard/cubic/release/view/head:/usr/share/cubic/utilities.py#L3012

[Cubic PPA (cubic-wizard)]

Ray,

Your issue is different than the issue Brian reported in this bug.

Looking again at your comment #2, I see that the output of...

    $ file /home/ubuntu/image/squashfs-root/boot/vmlinuz-4.18.0-rc2-udl+

..is producing a result that Cubic can not parse.

The output should look something like this example...

    $ sudo file /boot/vmlinuz-4.15.0-32-generic

Example output...

/boot/vmlinuz-4.15.0-32-generic: Linux kernel x86 boot executable bzImage, version 4.15.0-32-generic (buildd@lgw01-amd64-039) #35-Ubuntu SMP Fri Aug 10 17:58:07 UTC 2018, RO-rootFS, swap_dev 0x7, Normal VGA

Notice that the format of the version information ("version 4.15.0-32-generic") in this example is of the form:

    number.number.number-number

This is the format Cubic is looking for. Since you are compiling your own kernel, your version may have a different format than what Cubic expects.

If you can share the output of...

    $ file /home/ubuntu/image/squashfs-root/boot/vmlinuz-4.18.0-rc2-udl+

...I ~may~ be able to accommodate a generalization for your scenario in Cubic.

[Cubic PPA (cubic-wizard)]

William,

I customized cosmic-desktop-amd64.iso by adding add kernel 4.17.3-041703-generic from http://kernel.ubuntu.com/~kernel-ppa/mainline/.

I then selected kernel 4.17.3 as the ISO casper boot kernel in Cubic, and I was able to successfully boot the resulting ISO.

Note: I did use the development version of Cubic, 2018.08-171. I'll release this version in the next few days. However, I am still experiencing the original bug Brian reported when I try to customize a server version of Ubuntu.

[Cubic PPA (cubic-wizard)]

Used cubic to customize ubuntu-18.04-live-server-amd64.iso.

The kernel included in this image is 4.15.0-32-generic.

Upgraded to kernel 4.15.0-33-generic using...
    $ apt update
    $ apt install linux-image-generic

When I choose the default kernel, 4.15.0-32, I am able to boot the generated iso.
When I choose the upgraded kernel, 4.15.0-33, I get a kernel panic.

See the attached screenshot for the error.

[Cubic PPA (cubic-wizard)]

Add a notice on the Kernels tab to recommend that the user choose the default kernel to boot the live iso, if the user is customizing a server image.

Cubic guesses if the user is customizing a server image by checking if the original iso image filename, volume id, or disk name contain the word "server".

This "work-around" notice is available in Cubic release version 2018.09-39.

[Cubic PPA (cubic-wizard)]

Brian,

I just re-read your initial bug report. You stated: "Interestingly, if I select the 20 kernel and do the install, the new kernel is running on the resulting system"

This is normal/ correct behavior.

If you install a new kernel in the Cubic's chroot environment...
- The new kernel will be used inside your Live environment, launched from your customized ISO.
- The new kernel will be installed, if you install the OS from your customized ISO.

On Cubic's "Kernels" tab...
- You are ***NOT*** selecting which kernel your Live ISO uses.
- You are selecting which kernel is used by Casper to ~boot~ your live ISO.
- Theoretically, you could use any kernel to boot Casper,as long at it works.

When is this an issue?..
- If you install kernel modules or drivers in the chroot environment, but they are installed for a specific kernel
- In that case, those modules may not be available when you boot your Live ISO.
- An example may be Wi-Fi drivers. If they are installed for a different kernel than what you boot with, you may not have Wi-Fi access in your Live environment.
- Generally, this is not an issue.

TLDR...
- Leaving the default kernel should be good enough for most cases.
- You will still get to use your newly installed kernel in your Live ISO and in your OS installation.

[Cubic PPA (cubic-wizard)]

Set the default default bootstrap kernel in Cubic to be the newest available kernel. This resolves Bug #1825586, "Using the original ISO bootstrap kernel causes BusyBox, when that kernel version has been removed in the chroot environment."

Newer versions of Ubuntu do not exhibit the problem described in this bug report (Bug #1782379). Nevertheless, the user should select the original iso's bootstrap kernel if this issue occurs. (The original iso's bootstrap kernel is usually near the bottom of the kernels list presented by Cubic).