Ubuntu 19.10 - vmlinuz FileNotFoundError at 50% Update ISO Boot Files

[CtC]

Running Cubic:
    The Cubic version is................... 2019.11-59-release~201911270317~ubuntu18.04.1

Using the release version of Ubuntu:
    ubuntu-19.10-desktop-amd64.iso

After executing the following in the chroot environment:

sudo apt update
sudo apt -y upgrade

Then proceeding to write the custom ISO, the following error occurs:

  Update ISO boot files
    The selected kernel is index number.... 0
    Delete existig files with pattern..... /media/joseph/Haskell/cubic-
                                            error/custom-live-
                                            iso/casper/vmlinuz*
    Copy file.............................. Number 1 of 2
    The source file is..................... /media/joseph/Haskell/cubic-
                                            error/squashfs-root/boot/vmlinuz.old
    The target file is..................... /media/joseph/Haskell/cubic-
                                            error/custom-live-iso/casper/vmlinuz
    Execute asynchronously................. rsync --archive --no-relative --no-
                                            implied-dirs --info=progress2
                                            "/media/joseph/Haskell/cubic-
                                            error/squashfs-
                                            root/boot/vmlinuz.old"
                                            "/media/joseph/Haskell/cubic-
                                            error/custom-live-
                                            iso/casper/vmlinuz"
    Set a new process for thread id........ 139712253646592
    The new process id is.................. 20007
    • The start time is.................... 19:45:48.031257
    • Completed............................ 0%
    • Completed............................ 10%
    • Completed............................ 20%
    • Completed............................ 30%
    • Completed............................ 40%
    • Completed............................ 50%
    • Completed............................ 60%
    • Completed............................ 70%
    • Completed............................ 80%
    • Completed............................ 90%
    • Completed............................ 100%
    • The end time is...................... 19:45:49.125645
    Exception encountered in thread........ 139712253646592
    Ignore exception?...................... No
    The tracekback is...................... Traceback (most recent call last):
                                            File
                                            "/usr/share/cubic/transition.py",
                                            line 93, in run self.action()
                                            File
                                            "/usr/share/cubic/transition.py",
                                            line 436, in action self) File
                                            "/usr/share/cubic/transitions.py",
                                            line 2182, in transition__from__opti
                                            ons_page__to__repackage_iso_page
                                            utilities.copy_vmlinuz_and_initrd_fi
                                            les(thread) File
                                            "/usr/share/cubic/utilities.py",
                                            line 3260, in
                                            copy_vmlinuz_and_initrd_files
                                            stat.S_IRUSR | stat.S_IWUSR |
                                            stat.S_IRGRP | stat.S_IROTH)
                                            FileNotFoundError: [Errno 2] No such
                                            file or directory:
                                            '/media/joseph/Haskell/cubic-
                                            error/custom-live-
                                            iso/casper/vmlinuz'
    Finished running thread with id........ 139712253646592

This error does not occur with the release version of Ubuntu 18.04 LTS; A couple months back, I was able to create custom ISO images with Ubuntu 19.04.

Thank you for Cubic and the capabilities it provides! Please let me know if I missed crucial information.

[Cubic PPA (cubic-wizard)]

This issue doesn't happen if you don't execute..

    sudo apt update
    sudo apt -y upgrade

...in the chroot environment?

[CtC (joe-zx)]

Correct. I was able to generate a "custom ISO" based on the Ubuntu 19.10 ISO if I do not update and upgrade.

[Cubic PPA (cubic-wizard)]

I was able to reproduce this error using ubuntu-19.10-desktop-amd64.iso...

The tracekback is...................... Traceback (most recent call last):
                                        File
                                        "/usr/share/cubic/transition.py",
                                        line 93, in run self.action()
                                        File
                                        "/usr/share/cubic/transition.py",
                                        line 436, in action self) File
                                        "/usr/share/cubic/transitions.py",
                                        line 2182, in transition__from__opti
                                        ons_page__to__repackage_iso_page
                                        utilities.copy_vmlinuz_and_initrd_fi
                                        les(thread) File
                                        "/usr/share/cubic/utilities.py",
                                        line 3260, in
                                        copy_vmlinuz_and_initrd_files
                                        stat.S_IRUSR | stat.S_IWUSR |
                                        stat.S_IRGRP | stat.S_IROTH)
                                        FileNotFoundError: [Errno 2] No such
                                        file or directory:
                                        '/mnt/ram/Ubuntu-19.10/custom-live-
                                        iso/casper/vmlinuz'

[Cubic PPA (cubic-wizard)]

Steps:

(1)
Used: ubuntu-19.10-desktop-amd64.iso

(2)
Executed in chroot (No need for sudo, because user is already root!):
    apt update
    apt -y upgrade

(3)
45 combinations of vmlinuz and initrd were listed on the kernels tab in Cubic!
(This should have been 16, why 45???)

Accepted default option, i.e. the first option:
    vmlinuz.old
    init.img.old
    5.3.0-26
(The files vmlinuz.old and init.img.old, version 5.3.0-26, were also listed as the 2nd option. This is odd; they should be listed once only).

Clicked "Generate" button.

(4)
Encountered the error at 50% "update iso boot files". <--- ERROR !!!!

(5)
Clicked the back button.
Interestingly, this time I counted 59 combinations of vmlinuz and initrd.
(Was my original count incorrect, or did the number of options in the list actually change?)

(6)
Selected the last option in the list on the kernels tab:
    vmlinuz
    init.img
    5.3.0-18

Clicked "Generate" button.

(7)
This time, the FileNotFoundError error was NOT encountered.
ISO generation proceeded normally and was successful.

[Cubic PPA (cubic-wizard)]

CtC,

For now, if you are able, I suggest selecting the LAST option in the kernels list, on Cubic's Kernels tab. For Ubuntu 19.10, this is kernel 5.3.0-18 that is used to boot the original ISO from Canonical.

Please keep in mind, that the kernel used by your customized OS, once it is installed to a computer, will (usually) be the latest kernel that you had installed in the chroot environment. The purpose of the Kernels tab in Cubic is to select the kernel used to bootstrap the Linux OS from the ISO/USB. (Your computer needs to know how to read an OS from the ISO, and this is what bootsrtapping does).

Usually, it is best to select the kernel that Canonical has packages on the original ISO. However, one drawback is that, when you are in the Live environment of the customized ISO, it runs the kernel you had booted your ISO with; this is only for the Live environment, and not for the final installed system.

I do see an issue in the Kernels tab of Cubic, where way too many kernels are listed. However, even after this is fixed, please bear in mind, it is possible your customized ISO may not boot with an ISO boot kernel different from the one supplied by Canoncial (the official bootstrap kernel, last in the kernels list).

[Cubic PPA (cubic-wizard)]

Note: There are indeed 16 kernels listed + 1 original iso kernel. My origial count, by highlighting using the arrow keys, was incorrect.

[CtC (joe-zx)]

Previously, I had tried the last option in the list, and it did generate a custom ISO, but I was never able to boot using that image in Live mode. I have not tried to install the custom image (to the HD) as that is not my use case (see below).

So, to clarify, my use case for Cubic is to generate a custom ISO containing all the development tools and configuration I use, then to write the custom ISO to a flash drive. I modify the flash drive by reducing the partition used for the boot image (from the custom ISO), then creating a data partition in the remaining space. Finally, I use the custom flash drive to boot my laptops leaving Ubuntu in live mode. As I am often connecting to open wifi, I like the ability to run in live mode, and reformat the data partition as needed. The intent is to reduce the likelihood of a security breach, or to at least reduce the likelihood that malicious code would survive a reboot.

Now, if the custom image is using the older original kernel, it would lag behind with regards to security patches, if I am understanding correctly.

Again, thank you for the work on Cubic and for your assistance with this issue! Very much appreciated!

[Cubic PPA (cubic-wizard)]

Sometimes vmlinuz or initrd are simlinks to the actual files. Cubic presents these simlinks on the Boot Kernels tab, because the simlink names may be more recognizable to a user than the actual file names.

The issue occurs because Cubic copies the simlinks but does not copy the target files. This is not an issue if the user selects a row listing the actual target files on the Boot Kernels tab, because Cubic these files successfully.

For example "vmlinuz" may be a simlink to "vmlinuz-5.3.0-26-generic" and "initrd.img" may be a simlink to "initrd.img-5.3.0-26-generic".

If the user selects the boot kernel row that has both "vmlinuz-5.3.0-26-generic" and "initrd.img-5.3.0-26-generic", the boot kernel files will be copied successfully. However, if the use selects a boot kernel row that has either "vmlinuz" or "initrd.img", then the copy step fails at 50%.

A couple of options are available to fix this issue:
1. Always copy the actual target files instead of the simlink. (The files are renamed when they are copied, anyway).
2. Do not show simlinks in the boot kernels list. This option will reduce the number of kernels shown, making the UI less confusing. The user doesn’t necessarily have to recognize the file names, because the kernel versions are listed anyway.

[Cubic PPA (cubic-wizard)]

While investigating this bug, another issue was discovered: The list of vmlinuz and initrd files is larger than it should be. The actual list should contain 10 entries, but it actually contains 17 entries.

(The 10 entries are computed as follows: there are 3 vmlinuz files and there are 3 initrd files, all for the same kernel version, 5.3.0-26. Therefore there are 9 possible combinations (3x3=9) of these files the user could select. Finally, there is one vmlinuz and initrd file combination for version 5.3.0-18 from the original ISO. Therefore, there should be 9 + 1 = 10 possible kernel rows the user could chose from, not 16 + 1 = 17).

Cubic's algorithm that matches vmlinuz and initrd files, to list on the Boot Kernels tab, ends up adding more entries than are necessary.

One contributor to this issue is that, in Ubuntu 19.04, after an `apt update`, multiple simlinks are created, pointing to the same vmlinuz or initrd files. For example "vmlinuz" and "vmlinuz.old" both point to "vmlinuz-5.3.0-26-generic", and "initrd.img" and ""initrd.img.old" both point to "initrd.img-5.3.0-26-generic".

[Cubic PPA (cubic-wizard)]

Solution approach 2 was selected (see https://bugs.launchpad.net/cubic/+bug/1858376/comments/8).

vmlinuz and initrd simlinks will no longer be listed on the Kernels tab.

Additionally, the algorithm for matching vmlinuz and initrd was rewritten to eliminate the issue where redundant rows were being added to Cubic's UI on the Kernels tab (see https://bugs.launchpad.net/cubic/+bug/1858376/comments/9).

[Cubic PPA (cubic-wizard)]

Fix released to Development branch, revision 211.

[Cubic PPA (cubic-wizard)]

Fix released to Release branch, revision 60.

[CtC (joe-zx)]

The fix works perfect! I was able to build a custom ISO with all the latest updates and applications I need. Thank you!