depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'

[Paride Legovini]

[Impact]

* There is no impact to the end user but spurious error messages (tons of them) causing the perception that something really bad happened:

Example of one of those tons of messages:

depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'

[Test Case]

* sudo rm /lib/modules/$(uname -r)/modules.builtin.bin

* sudo depmod -a -F /boot/System.map-$(uname -r)

This will cause the spurious error messages because depmod -a -F won't have the "modules.builtin.bin" file created. This is a simulation of what the linux-modules postinst package does and an example of why the messages happen.

[Regression Potential]

* We are reverting a debian patch so we will be closer to upstream code base.

* the change is about 2 different priority changes on verbosity, so biggest problem would be extra (or fewer) stdout/stderr messages (for scripts relying on them, for example).

* I have tested with kernels 4.15, 5.0 and 5.3 and all of them seemed to work fine.

[Other Info]

* Original Description:

During a Focal install from the ISO image several errors like:

depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'

are logged in curtin's install logs. The installed system boots and works fine, but the error is clearly something we want to get rid of.

At first glance this seems related to:

https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1863261

but the version of initramfs-tools in both the installer and installed system (checked with `chroot /target dpkg -l initramfs-tools` during the installation) is 0.136ubuntu1, which should contain Rafael's fix for that bug. I wonder if the update-initramfs diversion has a role here.

I verified this happens at least on amd64 and arm64. I'm attaching the full install logs for a amd64 installation.
---
ProblemType: Bug
AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-14-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu18
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperVersion: 1.439
DistroRelease: Ubuntu 20.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
LiveMediaBuild: Ubuntu-Server 20.04 LTS "Focal Fossa" - Alpha amd64 (20200225)
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Lsusb-t:
 /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
 /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M
     |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 480M
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
Package: linux (not installed)
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: initrd=/casper/initrd quiet --- maybe-ubiquity
ProcVersionSignature: Ubuntu 5.4.0-14.17-generic 5.4.18
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-14-generic N/A
 linux-backports-modules-5.4.0-14-generic N/A
 linux-firmware 1.186
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.4.0-14-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.13.0-1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-4.2
dmi.modalias: dmi:bvnSeaBIOS:bvr1.13.0-1:bd04/01/2014:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-4.2:cvnQEMU:ct1:cvrpc-q35-4.2:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-4.2
dmi.sys.vendor: QEMU

[Ryan Harper (raharper)]

Is the reproducible not on the iso?

[Ryan Harper (raharper)]

Yes, I can see the error message in our FocalTestBasic.

The bug you mentioned specifies calls to update-initramfs, in curtin we're now diverting this binary, so it cannot be calls to update-initramfs. Instead looking at the patch in the bug you mention, it's /usr/sbin/mkinitramfs; which I suspect is being invoked *directly* in the package postinst.

I suspect curtin could divert this as well, however; it's not clear why we should do that; it appears to be an error within the package.

In focal, we have:

root@f1:~# apt-cache policy initramfs-tools-core
initramfs-tools-core:
  Installed: 0.133ubuntu14
  Candidate: 0.136ubuntu1
  Version table:
     0.136ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
 *** 0.133ubuntu14 100
        100 /var/lib/dpkg/status

However, the focal package does not have the change mentioned.

# grep -A1 -B1 "for x in modules" /usr/sbin/mkinitramfs
# Copy in modules.builtin and modules.order (not generated by depmod)
for x in modules.builtin modules.order; do
 if [ -f "${MODULESDIR}/${x}" ]; then

Vs. the fix:

+# and modules.builtin.bin (generated by depmod, but too late to avoid
+# error messages as in #948257)
+for x in modules.builtin modules.builtin.bin modules.order; do

[Ryan Harper (raharper)]

> However, the focal package does not have the change mentioned.

Of course 0.133ubuntu14 does not have the fix. I misread my own output.

[Ryan Harper (raharper)]

I can recreate the issue inside an LXC container on focal only (bionic, disco, eoan) and without any dpkg-divert of update-initramfs; as such I'm marking the curtin task invalid.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1864992

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Paride Legovini (paride)]

After some digging with Ryan and after trying with different kernel versions the problem seems due to a change in the kernel package. We didn't figure out which change in particular is causing the error.

[Ryan Harper (raharper)]

An easy recreate:

lxc launch ubuntu-daily:focal f1
lxc exec f1 bash
apt update && apt install linux-generic

This does not fail on eoan or bionic.

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Paride Legovini (paride)]

apport information

[Ryan Harper (raharper)]

Note, by fail, we mean depmod emits the error message mentioned in bug title; there is nothing *functionally* wrong; just scary/noisy output which it did not use to produce.

[Ryan Harper (raharper)]

I do not believe this is a duplicate; It is more likely a *packaging* issue. The question remains for this bug, why does it only appear in the focal kernels; but not Eoan or older?

And if someone could confirm whether or not the linux-generic or linux-modules generate this file and when (unpack, deb postinst, kernel install hook?)

Another reason why this isn't a duplicate as the error message is unrelated to the contents of the initrd, but rather when depmod -a -S is called (postins of linux-modules package).

On Eoan we can remove the file in question and regenerate the file with:

depmod -a -F /boot/System.map-5.3.0-40-generic;

this command does not produce the error message on any release expect Focal.

# eoan

root@rharper-e1:~# cat /etc/issue
Ubuntu 19.10 \n \l

root@rharper-e1:~# dpkg --list | grep linux-
ii linux-base 4.5ubuntu2 all Linux image base package
ii linux-firmware 1.183.4 all Firmware for Linux kernel drivers
ii linux-generic 5.3.0.40.34 amd64 Complete Generic Linux kernel and headers
ii linux-headers-5.3.0-40 5.3.0-40.32 all Header files related to Linux kernel version 5.3.0
ii linux-headers-5.3.0-40-generic 5.3.0-40.32 amd64 Linux kernel headers for version 5.3.0 on 64 bit x86 SMP
ii linux-headers-generic 5.3.0.40.34 amd64 Generic Linux kernel headers
ii linux-image-5.3.0-40-generic 5.3.0-40.32 amd64 Signed kernel image generic
ii linux-image-generic 5.3.0.40.34 amd64 Generic Linux kernel image
ii linux-modules-5.3.0-40-generic 5.3.0-40.32 amd64 Linux kernel extra modules for version 5.3.0 on 64 bit x86 SMP
ii linux-modules-extra-5.3.0-40-generic 5.3.0-40.32 amd64 Linux kernel extra modules for version 5.3.0 on 64 bit x86 SMP

root@rharper-e1:~# rm /lib/modules/5.3.0-40-generic/modules.builtin.bin
root@rharper-e1:~# depmod -a -F /boot/System.map-5.3.0-40-generic 5.3.0-40-generic
root@rharper-e1:~# ls -al /lib/modules/5.3.0-40-generic/modules.builtin.bin
-rw-r--r-- 1 root root 9943 Mar 2 15:03 /lib/modules/5.3.0-40-generic/modules.builtin.bin

# On focal this errors:
root@rharper-f1:~# cat /etc/issue
Ubuntu Focal Fossa (development branch) \n \l

root@rharper-f1:~# dpkg --list | grep linux-
ii linux-5.4-headers-5.4.0-14 5.4.0-14.17 all Header files related to Linux kernel version 5.4.0
ii linux-base 4.5ubuntu2 all Linux image base package
ii linux-firmware 1.186 all Firmware for Linux kernel drivers
ii linux-generic 5.4.0.14.17 amd64 Complete Generic Linux kernel and headers
ii linux-headers-5.4.0-14-generic ...

[Doug Smythies (dsmythies)]

I don't think this is duplicate either. I agree that this seems to be in the kernel, but don't yet have proof. If I compile the mainline kernel (5.6-rc4) myself i get this error 5600 times (once per module):

depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/home/doug/temp-k-git/linux/debian/linux-image/lib/modules/5.6.0-rc4-stock/modules.builtin.bin'

but the file is there (well, afterwards at least).
I didn't notice this with kernel 5.6-rc2, but might not have been looking.

[Kai-Heng Feng (kaihengfeng)]

Reproducible with kmod version 27-1ubuntu1. Previous version 26-1ubuntu1 doesn't have this issue.

[Doug Smythies (dsmythies)]

I have a 20.04 VM that I had not run for awhile (Feb 12th). It has kmod version 26-3ubuntu1, and installed mainline kernel 5.6.0-050600rc4-lowlatency fine.

The other two physical 20.04 test servers have kmod version 27-1ubuntu1, and both spew the error 5600 times during kernel installation the first time. If I reinstall the same kernel there are no errors spewed. If I purge the kernel and re-install it, then the error is spewed 5600 times again.

[Rafael David Tinoco (rafaeldtinoco)]

@dsmythies, @kaihengfeng

you seem to be correct.

First time I saw that issue was in a kmod regression test iirc.

The fix was supposed to be in:

initramfs-tools (0.133ubuntu15) focal; urgency=medium

from:

https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1863261

but I see now that might not cover all cases.

I'll revisit this today.

[Rafael David Tinoco (rafaeldtinoco)]

At that time .. kmod was 26+20191223-1ubuntu1, now kmod is 27-1ubuntu1... Will check if something happened in that upgrade that made the mkinitramfs change buggy OR its something that was never covered.

[Doug Smythies (dsmythies)]

For what it's worth, the versions of things in my test computers, all servers, no GUI:

name: serv-ff (a 20.04 VM, that has not had an update since around February 12th):
initramfs-tools: 0.133ubuntu14
kmod; 26-3ubuntu1

name: s18 (a physical i5-9600K based computer, up to date as of a few days ago):
initramfs-tools: 0.136ubuntu1
kmod; 27-1ubuntu1

name: s15 (a physical i7-2600K based computer, up to date as of a few days ago):
initramfs-tools: 0.136ubuntu1
kmod; 27-1ubuntu1

[corrado venturini (corradoventu)]

I have a similar problem: more than 4000 messages:
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.6.0-050600rc4-generic/modules.builtin.bin'
installing kernel 5.6.0-050600rc4-generic downloadsd from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.6-rc4/

[Juerg Haefliger (juergh)]

There seem to be multiple issues related to these benign errors:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=948257
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=949444

Just to be clear, it's not a kernel (linux) package problem.

[Rafael David Tinoco (rafaeldtinoco)]

Related bugs:

https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1863261
https://bugs.launchpad.net/ubuntu/+source/kmod/+bug/1866012
https://bugs.launchpad.net/ubuntu/+source/kmod/+bug/1864992

[Rafael David Tinoco (rafaeldtinoco)]

Alright, afaict what is happening is this:

In Focal I first faced this issue when upgrading kmod to 26+20191223-1ubuntu1.. but my reproducer was "update-initramfs" only. The issue is that both: mkinitramfs (initramfs-tools) needs a patch to work with recent 5.x kernels:

# mkinitramfs (initramfs-tools)

# Copy in modules.builtin and modules.order (not generated by depmod)
# and modules.builtin.bin (generated by depmod, but too late to avoid
# error messages as in #948257)
for x in modules.builtin modules.builtin.bin modules.order; do
        if [ -f "${MODULESDIR}/${x}" ]; then
                cp -p "${MODULESDIR}/${x}" "${DESTDIR}${MODULESDIR}/${x}"
        fi
done

All other Ubuntu versions that might use a HWE kernel will need a SRU of that patch.

But thats not the only way to trigger the issue as it seems...

I was able to reproduce the issue when doing:

(k)rafaeldtinoco@kmodissuefocal:~$ apt-get install linux-image-5.4.0-14-generic linux-headers-5.4.0-14-generic linux-modules-extra-5.4.0-14-generic

while I had kernel 5.4.0-18-generic installed.

Setting up linux-modules-5.4.0-14-generic (5.4.0-14.17) ...
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-14-generic/modules.builtin.bin'

The same happened when installing linux-modules-5.4.0-9-generic AND when uninstalling and re-installing 5.4.0-18-generic. So now I'll reproduce this second issue.. and I'm marking LP: #1863261 that SRU is needed for that case.

For this case.. it turns out that postinst script for linux-modules is executing:

$ sudo depmod -a -F /boot/System.map-5.4.0-18-generic 5.4.0-18-generic

for the first time when /lib/modules/5.4.0-18-generic/modules.builtin.bin DOES NOT exist yet.

an easy way to reproduce is this:

$ sudo rm /lib/modules/5.4.0-18-generic/modules.builtin.bin

and then:

$ sudo depmod -a -F /boot/System.map-5.4.0-18-generic 5.4.0-18-generic
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-18-generic/modules.builtin.bin'
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-18-generic/modules.builtin.bin'
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-18-generic/modules.builtin.bin'
depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-18-generic/modules....

[Rafael David Tinoco (rafaeldtinoco)]

Kmod source shows:

idx = index_file_open(fn);
if (idx == NULL) {
    DBG(ctx, "could not open builtin file '%s'\n", fn);
    return NULL;
}

and this is continuously called.

I'll check if DBG should be enabled in our case OR we should patch upstream.

[Rafael David Tinoco (rafaeldtinoco)]

Here is the thing...

log_setup_kmod_log() is the function controlling verbocity level. When called from do_modprobe() execution path, it has a way to control the "log priority", being able to control how verbose the execution should be. Meanwhile, when log_setup_kmod_log() is called from do_depmod().. it has no way to control how verbose the rest of execution will be.

#define DEFAULT_VERBOSE LOG_WARNING
static int verbose = DEFAULT_VERBOSE;

and then:

depmod():
...
case 'v':
  verbose++;
  break;
...
  log_setup_kmod_log(ctx, verbose);
...

since we have:

#define LOG_EMERG 0 /* system is unusable */
#define LOG_ALERT 1 /* action must be taken immediately */
#define LOG_CRIT 2 /* critical conditions */
#define LOG_ERR 3 /* error conditions */
#define LOG_WARNING 4 /* warning conditions */
#define LOG_NOTICE 5 /* normal but significant condition */
#define LOG_INFO 6 /* informational */
#define LOG_DEBUG 7 /* debug-level messages *

Something is bumping the verbose from "warning" into "debug" automatically.

[Rafael David Tinoco (rafaeldtinoco)]

@cascardo brought to us that there is on-going discussion about this in:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=948257#95

which I missed...

Debian has this patch:

+ debian/patches/verbose_missing_bin

Description: Report an error when some .bin files do not exist
Author: Marco d'Itri <email address hidden>
Bug-Debian: http://bugs.debian.org/684901
---

--- a/libkmod/libkmod.c
+++ b/libkmod/libkmod.c
@@ -503,7 +503,7 @@ static char *lookup_builtin_file(struct

                idx = index_file_open(fn);
                if (idx == NULL) {
- DBG(ctx, "could not open builtin file '%s'\n", fn);
+ ERR(ctx, "could not open builtin file '%s'\n", fn);
                        return NULL;
                }

@@ -575,7 +575,7 @@ char *kmod_search_moddep(struct kmod_ctx

        idx = index_file_open(fn);
        if (idx == NULL) {
- DBG(ctx, "could not open moddep file '%s'\n", fn);
+ ERR(ctx, "could not open moddep file '%s'\n", fn);
                return NULL;
        }

Changing DBG to ERR... that changes the involved print logic.

I'm still playing with this as I want to check what is changing verbosity when execution path comes from depmod.

[Thadeu Lima de Souza Cascardo (cascardo)]

https://<email address hidden>/T/#u

This patch just caught my attention. I didn't do any investigation, but as I recall seeing something about namespaces, maybe this is related.

Cascardo.

[Rafael David Tinoco (rafaeldtinoco)]

Sorry @cascardo, forgot to update there. I'll revert half of Debian patch... (the could not open builtin file).. turning it back to DBG() only message. This "fixes" the issue. We can solve implement whatever Debian decides later.. but getting rid of those messages now is good or we will have many bugs popping up. Let me suggest a small MR for this.

[Rafael David Tinoco (rafaeldtinoco)]

Reason for keeping verbose_missing_bin patch was because of test case described at:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=684901

(k)root@fkmodissue:.../5.4.0-18-generic$ sudo mv modules.alias.bin modules.alias.bin.old

(k)root@fkmodissue:.../5.4.0-18-generic$ sudo modprobe hehe
modprobe: FATAL: Module hehe not found in directory /lib/modules/5.4.0-18-generic

and it doesn't apply any longer with kernels 5.4 in focal.

Testing Bionic for the same test case:

(k)rafaeldtinoco@bkmodissue:.../5.3.0-42-generic$ sudo mv modules.alias.bin modules.alias.bin.old

(k)rafaeldtinoco@bkmodissue:.../5.3.0-42-generic$ sudo modprobe hehe
modprobe: FATAL: Module hehe not found in directory /lib/modules/5.3.0-42-generic

and with 5.0 kernels:

(k)rafaeldtinoco@bkmodissue:.../5.0.0-43-generic$ sudo mv modules.alias.bin modules.alias.bin.old

(k)rafaeldtinoco@bkmodissue:.../5.0.0-43-generic$ sudo modprobe hehe
modprobe: FATAL: Module hehe not found in directory /lib/modules/5.0.0-43-generic

I think its safe to drop it since that Debian bug (from 2012) doesn't exist today.

Will re-do this test with patch dropped to make sure we solve this issue and don't cause regression in to that one.

[Rafael David Tinoco (rafaeldtinoco)]

@cascardo did a MR for focal and added you as a reviewer. will do the other MR for eoan and bionic now.

[Rafael David Tinoco (rafaeldtinoco)]

I have created a PPA for this issue:

https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1864992

Please provide feedback if possible. I haven't done Focal yet because git-ubuntu is facing a small problem importing latest focal source package to do the merge.. will check it tomorrow.

[Rafael David Tinoco (rafaeldtinoco)]

Testing Bionic kernel with the initial situation the bug being dropped said to fix:

(k)rafaeldtinoco@bkmodissue:~$ sudo rm /lib/modules/4.15.0-91-generic/modules.builtin.bin
(k)rafaeldtinoco@bkmodissue:~$ sudo depmod -a -F /boot/System.map-4.15.0-91-generic

ok

(k)rafaeldtinoco@bkmodissue:.../4.15.0-91-generic$ sudo mv modules.alias.bin modules.alias.bin.old

(k)rafaeldtinoco@bkmodissue:.../4.15.0-91-generic$ sudo modprobe hehe
modprobe: FATAL: Module hehe not found in directory /lib/modules/4.15.0-91-generic

and it is also safe to drop it based on previous bug test case

[Launchpad Janitor (janitor)]

This bug was fixed in the package kmod - 27-1ubuntu2

---------------
kmod (27-1ubuntu2) focal; urgency=medium

  * d/p/verbose_missing_bin: drop this patch to avoid spurious missing
    file messages when running depmod -a for the first time in recent
    kernels (>= 5.3.x) (LP: #1864992)

 -- Rafael David Tinoco <email address hidden> Thu, 12 Mar 2020 13:15:28 +0000

[Timo Aaltonen (tjaalton)]

Hello Paride, or anyone else affected,

Accepted kmod into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/kmod/26-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Paride, or anyone else affected,

Accepted kmod into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/kmod/24-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted kmod (26-1ubuntu1.1) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

cross-toolchain-base-mipsen/unknown (armhf)
systemd/242-7ubuntu3.7 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#kmod

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted kmod (24-1ubuntu3.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/237-3ubuntu10.39 (i386)
linux-hwe-5.0/5.0.0-44.48~18.04.1 (armhf)
ipset/6.34-1 (ppc64el, s390x, amd64, i386, arm64)
chrony/3.2-4ubuntu4.2 (ppc64el, s390x, amd64, armhf, i386, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#kmod

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Matthias Klose (doko)]

reopening for focal. still seen in almost every autopkg test (2020-03-24)

[Rafael David Tinoco (rafaeldtinoco)]

Thanks doko, focal was Fix Released for the original initramfs-tools, iirc. That made me miss the fix for Focal. I'm applying the same fix (Eoan/Bionic SRUs) to Focal.

[Rafael David Tinoco (rafaeldtinoco)]

Actually, I missed my previous fix:

---
This bug was fixed in the package kmod - 27-1ubuntu2

---------------
kmod (27-1ubuntu2) focal; urgency=medium

  * d/p/verbose_missing_bin: drop this patch to avoid spurious missing
    file messages when running depmod -a for the first time in recent
    kernels (>= 5.3.x) (LP: #1864992)

 -- Rafael David Tinoco <email address hidden> Thu, 12 Mar 2020 13:15:28 +0000
---

and it should have taken care of this issue.. checking the given example:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/d/doit/20200323_182614_c1613@/log.gz

It turns out the kernel gets updated *before* kmod fix.. and that triggers the buggy logic. For that not to happen we would have to have an updated cloud-image (with kmod 27-1ubuntu2) or update kmod before the kernel.

Marking it back to Fix Released, feel free to re-open it if you think this is wrong.

[Rafael David Tinoco (rafaeldtinoco)]

> Hello Paride, or anyone else affected,
>
> Accepted kmod into eoan-proposed. The package will build now and be
> available at https://launchpad.net/ubuntu/+source/kmod/26-1ubuntu1.1 in
> a few hours, and then in the -proposed repository.

Verification for eoan-proposed:

(k)rafaeldtinoco@ekmodissue:~$ apt changelog kmod
Get:1 https://changelogs.ubuntu.com kmod 26-1ubuntu1.1 Changelog [21.1 kB]
Fetched 21.1 kB in 2s (10.4 kB/s)

kmod (26-1ubuntu1.1) eoan; urgency=medium

  * d/p/verbose_missing_bin: drop this patch to avoid spurious missing
    file messages when running depmod -a for the first time in recent
    kernels (>= 5.3.x) (LP: #1864992)
  * Removed --add-udeb from dh_makeshlibs, since libkmod2-udeb does not
    actually contain a library. (Closes: #939779)

 -- Rafael David Tinoco <email address hidden> Wed, 11 Mar 2020 19:32:52 +0000

(k)rafaeldtinoco@ekmodissue:~$ apt-cache policy kmod
kmod:
  Installed: 26-1ubuntu1.1
  Candidate: 26-1ubuntu1.1
  Version table:
 *** 26-1ubuntu1.1 500
        500 http://us.archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     26-1ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu eoan/main amd64 Packages

(k)rafaeldtinoco@ekmodissue:~$ uname -a
Linux ekmodissue 5.3.0-43-generic #36-Ubuntu SMP Mon Mar 16 13:29:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
(k)rafaeldtinoco@ekmodissue:~$ sudo rm /lib/modules/$(uname -r)/modules.builtin.bin
(k)rafaeldtinoco@ekmodissue:~$ sudo depmod -a -F /boot/System.map-$(uname -r)
(k)rafaeldtinoco@ekmodissue:~$

[Rafael David Tinoco (rafaeldtinoco)]

> Hello Paride, or anyone else affected,
>
> Accepted kmod into bionic-proposed. The package will build now and be
> available at https://launchpad.net/ubuntu/+source/kmod/24-1ubuntu3.3 in
> a few hours, and then in the -proposed repository.

kmod (24-1ubuntu3.3) bionic; urgency=medium

  * d/p/verbose_missing_bin: drop this patch to avoid spurious missing
    file messages when running depmod -a for the first time in recent
    kernels (>= 5.3.x) (LP: #1864992)
  * Removed --add-udeb from dh_makeshlibs, since libkmod2-udeb does not
    actually contain a library. (Closes: #939779)

 -- Rafael David Tinoco <email address hidden> Wed, 11 Mar 2020 19:37:06 +0000

(k)rafaeldtinoco@bkmodissue:~$ apt-cache policy kmod
kmod:
  Installed: 24-1ubuntu3.3
  Candidate: 24-1ubuntu3.3
  Version table:
 *** 24-1ubuntu3.3 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     24-1ubuntu3.2 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     24-1ubuntu3 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

(k)rafaeldtinoco@bkmodissue:~$ uname -a
Linux bkmodissue 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri Feb 28 13:42:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
(k)rafaeldtinoco@bkmodissue:~$ sudo rm /lib/modules/$(uname -r)/modules.builtin.bin
(k)rafaeldtinoco@bkmodissue:~$ sudo depmod -a -F /boot/System.map-$(uname -r)
(k)rafaeldtinoco@bkmodissue:~$

(k)rafaeldtinoco@bkmodissue:~$ uname -a
Linux bkmodissue 5.0.0-43-generic #47~18.04.1-Ubuntu SMP Mon Mar 2 04:28:21 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
(k)rafaeldtinoco@bkmodissue:~$ sudo rm /lib/modules/$(uname -r)/modules.builtin.bin
(k)rafaeldtinoco@bkmodissue:~$ sudo depmod -a -F /boot/System.map-$(uname -r)
(k)rafaeldtinoco@bkmodissue:~$

[Launchpad Janitor (janitor)]

This bug was fixed in the package kmod - 26-1ubuntu1.1

---------------
kmod (26-1ubuntu1.1) eoan; urgency=medium

  * d/p/verbose_missing_bin: drop this patch to avoid spurious missing
    file messages when running depmod -a for the first time in recent
    kernels (>= 5.3.x) (LP: #1864992)
  * Removed --add-udeb from dh_makeshlibs, since libkmod2-udeb does not
    actually contain a library. (Closes: #939779)

 -- Rafael David Tinoco <email address hidden> Wed, 11 Mar 2020 19:32:52 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for kmod has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package kmod - 24-1ubuntu3.3

---------------
kmod (24-1ubuntu3.3) bionic; urgency=medium

  * d/p/verbose_missing_bin: drop this patch to avoid spurious missing
    file messages when running depmod -a for the first time in recent
    kernels (>= 5.3.x) (LP: #1864992)
  * Removed --add-udeb from dh_makeshlibs, since libkmod2-udeb does not
    actually contain a library. (Closes: #939779)

 -- Rafael David Tinoco <email address hidden> Wed, 11 Mar 2020 19:37:06 +0000