systemd-modules-load.service: Failing due to missing module 'ib_iser'

[Christian Sarrasin]

On a number of VMs using Xenial daily builds 20160330 and 20160403, out of the box (just after booting for the first time), the following appears in dmesg at boot time:

systemd[1]: systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start Load Kernel Modules.
systemd[1]: systemd-modules-load.service: Unit entered failed state.
systemd[1]: systemd-modules-load.service: Failed with result 'exit-code'.

The following related complaint appears in /var/log/syslog:

ubuntu systemd-modules-load[352]: Inserted module 'iscsi_tcp'
ubuntu systemd-modules-load[352]: Failed to find module 'ib_iser'

It looks like 'ib_iser' is some Mellanox-related module, which I don't *think* is something that a stock cloudimg install should attempt to load but I may be wrong; in any case, it's not present (nor is there any similar whingeing) on similar VMs running off 14.04 cloudimg.

More info: http://askubuntu.com/questions/753672/xenial-beta2-systemd-modules-load-service-failed-due-to-missing-module-ib-i

---
root@testvm4:~# lsb_release -rd
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04
root@testvm4:~# apt policy systemd
systemd:
  Installed: 229-3ubuntu2
  Candidate: 229-3ubuntu2

[Martin Pitt (pitti)]

Well, don't shoot the messenger :-) ib_iser.ko is in linux-image-extra which we (consciously) don't install on cloud images. So please fix the cloud image builds or cloud-init (whichever it is) to not add the module to /etc/modules by default, unless you actually install linux-image-extra.

[Christian Sarrasin (sxc731)]

Didn't mean to blame systemd-modules-load.service specifically; as you point out, it's just where the smoking gun appears. I suppose the only thing that's debatable is whether it should fail in these circumstances and whether it would inhibit other, unrelated, modules to load? (that would be bad)

Also FWIW, as noted on the investigations in the 'ask' question, /etc/modules-load.d/modules.conf is empty as is /usr/lib/modules-load.d. I haven't managed to work out what is triggering this particular module to be considered for loading. A grep for 'ib_iser' in /var/log reveals nothing beside what's in the report.

[Scott Moser (smoser)]

Likely related/result of bug 1115710.
The reference to mlx4.conf is what lead me to believe that. I'll dig a bit more tomorrow.

[Christian Sarrasin (sxc731)]

That mlx4.conf file is also present in the same form on 14.04 cloudimg, which doesn't suffer from the issue. Or maybe the systemd module loader interprets this differently?

BTW the workaround I've found is to add the following to /etc/modprobe.d/blacklist.conf:

blacklist ib_iser
install ib_iser /bin/false

[Scott Moser (smoser)]

ok, so it is /lib/modules-load.d/open-iscsi.conf that has 'ib_iser' in it.

ib_iser per modinfo "iSER (iSCSI Extensions for RDMA) Datamover".

per depmod on full linux-image-extra system:
 kernel/drivers/infiniband/ulp/iser/ib_iser.ko: kernel/drivers/infiniband/core/rdma_cm.ko kernel/drivers/infiniband/core/iw_cm.ko kernel/drivers/infiniband/core/ib_cm.ko kernel/drivers/infiniband/core/ib_sa.ko kernel/drivers/infiniband/core/ib_mad.ko kernel/drivers/infiniband/core/ib_core.ko kernel/drivers/infiniband/core/ib_addr.ko kernel/drivers/scsi/libiscsi.ko kernel/drivers/scsi/scsi_transport_iscsi.ko

$ ls -l /lib/modules/4.4.0-17-generic/kernel/drivers/infiniband/ulp/iser/ib_iser.ko
-rw-r--r-- 1 root root 81382 Mar 29 15:15 /lib/modules/4.4.0-17-generic/kernel/drivers/infiniband/ulp/iser/ib_iser.ko

So the easiest fix for this, at a cost of 81kb is to just move ib_iser from linux-image-extra to linux-image.

There might be some way to say "load if present, do not complain" but I dont see it immediately. Possibly something with modprobe.d. It would seem that such function would be needed to avoid warning failure if a driver was simply built into the kernel. Ie, a custom kernel built ib_iser into the kernel, then we'd not want to warn.

[Scott Moser (smoser)]

So, some conversation with infinity at http://irclogs.ubuntu.com/2016/04/07/%23ubuntu-devel.html .
All of ib_iser's deps are already in linux-generic (http://paste.ubuntu.com/15670921/)

So we might as well move that module also.

[Martin Pitt (pitti)]

> So the easiest fix for this, at a cost of 81kb is to just move ib_iser from linux-image-extra to linux-image.

It actually appears to me that the easier and less risky (as in "minimize behaviour change") fix is to just drop /lib/modules-load.d/open-iscsi.conf ? I mean, the module hasn't been loaded until now, so it can hardly be critical?

[Adam Conrad (adconrad)]

It's loading the first module, then dying. Pretty sure we want open-iscsi.conf to continue existing, and in light of all the iscsi and infiniband scaffolding being in the virtual image, it seems silly to exclude the one module that glues them together.

[Tim Gardner (timg-tpi)]

I've added ib_iser and ib_isert to linux-image

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-21.37

---------------
linux (4.4.0-21.37) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1571791

  * linux: MokSBState is ignored (LP: #1571691)
    - SAUCE: (noup) MODSIGN: Import certificates from UEFI Secure Boot
    - SAUCE: (noup) efi: Disable secure boot if shim is in insecure mode
    - SAUCE: (noup) Display MOKSBState when disabled

linux (4.4.0-20.36) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1571069

  * sysfs mount failure during stateful lxd snapshots (LP: #1570906)
    - SAUCE: kernfs: Do not match superblock in another user namespace when
      mounting

  * Kernel Panic in Ubuntu 16.04 netboot installer (LP: #1570441)
    - x86/topology: Fix logical package mapping
    - x86/topology: Fix Intel HT disable
    - x86/topology: Use total_cpus not nr_cpu_ids for logical packages
    - xen/apic: Provide Xen-specific version of cpu_present_to_apicid APIC op
    - x86/topology: Fix AMD core count

  * [regression]: Failed to call clock_adjtime(): Invalid argument
    (LP: #1566465)
    - ntp: Fix ADJ_SETOFFSET being used w/ ADJ_NANO

linux (4.4.0-19.35) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1570348

  * CVE-2016-2847 (LP: #1554260)
    - pipe: limit the per-user amount of pages allocated in pipes

  * xenial kernel crash on HP BL460c G7 (qla24xx problem?) (LP: #1554003)
    - SAUCE: (noup) qla2xxx: Add irq affinity notification V2

  * arm64: guest hangs when ntpd is running (LP: #1549494)
    - SAUCE: (noup) KVM: arm/arm64: Handle forward time correction gracefully

  * linux: Enforce signed module loading when UEFI secure boot (LP: #1566221)
    - [Config] CONFIG_EFI_SECURE_BOOT_SIG_ENFORCE=y

  * s390/cpumf: Fix lpp detection (LP: #1555344)
    - s390/facilities: use stfl mnemonic instead of insn magic
    - s390/facilities: always use lowcore's stfle field for storing facility bits
    - s390/cpumf: Fix lpp detection

  * s390x kernel image needs weightwatchers (LP: #1536245)
    - [Config] s390x: Use compressed kernel bzImage

  * Surelock GA2 SP1: surelock02p05: Not seeing sgX devices for LUNs after
    upgrading to Ubuntu 16.04 (LP: #1567581)
    - Revert "UBUNTU: SAUCE: (noup) powerpc/pci: Assign fixed PHB number based on
      device-tree properties"

  * Backport upstream bugfixes to ubuntu-16.04 (LP: #1555765)
    - cpufreq: powernv: Define per_cpu chip pointer to optimize hot-path
    - Revert "cpufreq: postfix policy directory with the first CPU in related_cpus"
    - cpufreq: powernv: Add sysfs attributes to show throttle stats

  * systemd-modules-load.service: Failing due to missing module 'ib_iser' (LP: #1566468)
    - [Config] Add ib_iser to generic inclusion list

  * thunderx nic performance improvements (LP: #1567093)
    - net: thunderx: Set recevie buffer page usage count in bulk
    - net: thunderx: Adjust nicvf structure to reduce cache misses

  * fixes for thunderx nic in multiqueue mode (LP: #1567091)
    - net: thunderx: Fix for multiqset not configured upon interface toggle
    - net: thunderx: Fix for HW TSO not enabled for secondary qsets
    - net: thund...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[HITMAn (hitmansoft)]

I've noticed the same problem on official ubuntu server 16.04.1 image for raspi2 and marked this bug as also affecting linux-raspi2 package.

[Simon Kropf (simon-kropf)]

Same bug here on ubuntu server 16.04 raspi3 image.

Install of pptpd via apt-get failed.
Fixed tempoarily -> ib_iser to #ib_iser in /lib/modules-load.d/open-iscsi.conf

[Simon Wydooghe (wydooghe-simon)]

Confirmed on rpi2 image. Removing open-iscsi package fixes the 'problem' for me.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1566468

[Adam Smith (adamsmith)]

I've created a new bug for the raspi2 server image - Bug #1765154

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Dodd (anarchetic)]

FWIW, this still seems to be a problem with Ubuntu Minimal cloud images for bionic? I'm using them in a slightly non-standard configuration (ripping the root file system image out and handing it to qemu), but this does seem to be a problem with the image itself - the modules are included on the root filesystem, not including ib_iser, but the relevant snippet calling for the module is.

[Paul Larson (pwlars)]

This also shows up still in the 18.04.5 candidate server images on raspberry pi