systemd-modules-load.service: Failing due to missing module 'ib_iser'

Bug #1566468 reported by Christian Sarrasin on 2016-04-05
26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Medium
Tim Gardner
Nominated for Bionic by Dan Watkins
Xenial
Medium
Tim Gardner
linux-kvm (Ubuntu)
Undecided
Unassigned
Nominated for Bionic by Dan Watkins
Xenial
Undecided
Unassigned
linux-raspi2 (Ubuntu)
Undecided
Unassigned
Nominated for Bionic by Dan Watkins
Xenial
Undecided
Unassigned
open-iscsi (Ubuntu)
Medium
Unassigned
Nominated for Bionic by Dan Watkins
Xenial
Medium
Unassigned

Bug Description

On a number of VMs using Xenial daily builds 20160330 and 20160403, out of the box (just after booting for the first time), the following appears in dmesg at boot time:

systemd[1]: systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start Load Kernel Modules.
systemd[1]: systemd-modules-load.service: Unit entered failed state.
systemd[1]: systemd-modules-load.service: Failed with result 'exit-code'.

The following related complaint appears in /var/log/syslog:

ubuntu systemd-modules-load[352]: Inserted module 'iscsi_tcp'
ubuntu systemd-modules-load[352]: Failed to find module 'ib_iser'

It looks like 'ib_iser' is some Mellanox-related module, which I don't *think* is something that a stock cloudimg install should attempt to load but I may be wrong; in any case, it's not present (nor is there any similar whingeing) on similar VMs running off 14.04 cloudimg.

More info: http://askubuntu.com/questions/753672/xenial-beta2-systemd-modules-load-service-failed-due-to-missing-module-ib-i

---
root@testvm4:~# lsb_release -rd
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04
root@testvm4:~# apt policy systemd
systemd:
  Installed: 229-3ubuntu2
  Candidate: 229-3ubuntu2

CVE References

Martin Pitt (pitti) wrote :

Well, don't shoot the messenger :-) ib_iser.ko is in linux-image-extra which we (consciously) don't install on cloud images. So please fix the cloud image builds or cloud-init (whichever it is) to not add the module to /etc/modules by default, unless you actually install linux-image-extra.

affects: systemd (Ubuntu) → cloud-init (Ubuntu)
Christian Sarrasin (sxc731) wrote :

Didn't mean to blame systemd-modules-load.service specifically; as you point out, it's just where the smoking gun appears. I suppose the only thing that's debatable is whether it should fail in these circumstances and whether it would inhibit other, unrelated, modules to load? (that would be bad)

Also FWIW, as noted on the investigations in the 'ask' question, /etc/modules-load.d/modules.conf is empty as is /usr/lib/modules-load.d. I haven't managed to work out what is triggering this particular module to be considered for loading. A grep for 'ib_iser' in /var/log reveals nothing beside what's in the report.

Scott Moser (smoser) wrote :

Likely related/result of bug 1115710.
The reference to mlx4.conf is what lead me to believe that. I'll dig a bit more tomorrow.

Christian Sarrasin (sxc731) wrote :

That mlx4.conf file is also present in the same form on 14.04 cloudimg, which doesn't suffer from the issue. Or maybe the systemd module loader interprets this differently?

BTW the workaround I've found is to add the following to /etc/modprobe.d/blacklist.conf:

blacklist ib_iser
install ib_iser /bin/false

Scott Moser (smoser) on 2016-04-07
affects: cloud-init (Ubuntu) → open-iscsi (Ubuntu)
Changed in open-iscsi (Ubuntu):
importance: Undecided → Medium
status: New → Confirmed
Scott Moser (smoser) wrote :

ok, so it is /lib/modules-load.d/open-iscsi.conf that has 'ib_iser' in it.

ib_iser per modinfo "iSER (iSCSI Extensions for RDMA) Datamover".

per depmod on full linux-image-extra system:
 kernel/drivers/infiniband/ulp/iser/ib_iser.ko: kernel/drivers/infiniband/core/rdma_cm.ko kernel/drivers/infiniband/core/iw_cm.ko kernel/drivers/infiniband/core/ib_cm.ko kernel/drivers/infiniband/core/ib_sa.ko kernel/drivers/infiniband/core/ib_mad.ko kernel/drivers/infiniband/core/ib_core.ko kernel/drivers/infiniband/core/ib_addr.ko kernel/drivers/scsi/libiscsi.ko kernel/drivers/scsi/scsi_transport_iscsi.ko

$ ls -l /lib/modules/4.4.0-17-generic/kernel/drivers/infiniband/ulp/iser/ib_iser.ko
-rw-r--r-- 1 root root 81382 Mar 29 15:15 /lib/modules/4.4.0-17-generic/kernel/drivers/infiniband/ulp/iser/ib_iser.ko

So the easiest fix for this, at a cost of 81kb is to just move ib_iser from linux-image-extra to linux-image.

There might be some way to say "load if present, do not complain" but I dont see it immediately. Possibly something with modprobe.d. It would seem that such function would be needed to avoid warning failure if a driver was simply built into the kernel. Ie, a custom kernel built ib_iser into the kernel, then we'd not want to warn.

Scott Moser (smoser) wrote :

So, some conversation with infinity at http://irclogs.ubuntu.com/2016/04/07/%23ubuntu-devel.html .
All of ib_iser's deps are already in linux-generic (http://paste.ubuntu.com/15670921/)

So we might as well move that module also.

Changed in linux (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Tim Gardner (timg-tpi) on 2016-04-07
Changed in linux (Ubuntu Xenial):
assignee: nobody → Tim Gardner (timg-tpi)
status: Confirmed → In Progress
Tim Gardner (timg-tpi) on 2016-04-07
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Martin Pitt (pitti) wrote :

> So the easiest fix for this, at a cost of 81kb is to just move ib_iser from linux-image-extra to linux-image.

It actually appears to me that the easier and less risky (as in "minimize behaviour change") fix is to just drop /lib/modules-load.d/open-iscsi.conf ? I mean, the module hasn't been loaded until now, so it can hardly be critical?

Adam Conrad (adconrad) wrote :

It's loading the first module, then dying. Pretty sure we want open-iscsi.conf to continue existing, and in light of all the iscsi and infiniband scaffolding being in the virtual image, it seems silly to exclude the one module that glues them together.

Tim Gardner (timg-tpi) wrote :

I've added ib_iser and ib_isert to linux-image

Adam Conrad (adconrad) on 2016-04-07
Changed in open-iscsi (Ubuntu Xenial):
status: Confirmed → Invalid
Launchpad Janitor (janitor) wrote :
Download full text (3.5 KiB)

This bug was fixed in the package linux - 4.4.0-21.37

---------------
linux (4.4.0-21.37) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1571791

  * linux: MokSBState is ignored (LP: #1571691)
    - SAUCE: (noup) MODSIGN: Import certificates from UEFI Secure Boot
    - SAUCE: (noup) efi: Disable secure boot if shim is in insecure mode
    - SAUCE: (noup) Display MOKSBState when disabled

linux (4.4.0-20.36) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1571069

  * sysfs mount failure during stateful lxd snapshots (LP: #1570906)
    - SAUCE: kernfs: Do not match superblock in another user namespace when
      mounting

  * Kernel Panic in Ubuntu 16.04 netboot installer (LP: #1570441)
    - x86/topology: Fix logical package mapping
    - x86/topology: Fix Intel HT disable
    - x86/topology: Use total_cpus not nr_cpu_ids for logical packages
    - xen/apic: Provide Xen-specific version of cpu_present_to_apicid APIC op
    - x86/topology: Fix AMD core count

  * [regression]: Failed to call clock_adjtime(): Invalid argument
    (LP: #1566465)
    - ntp: Fix ADJ_SETOFFSET being used w/ ADJ_NANO

linux (4.4.0-19.35) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1570348

  * CVE-2016-2847 (LP: #1554260)
    - pipe: limit the per-user amount of pages allocated in pipes

  * xenial kernel crash on HP BL460c G7 (qla24xx problem?) (LP: #1554003)
    - SAUCE: (noup) qla2xxx: Add irq affinity notification V2

  * arm64: guest hangs when ntpd is running (LP: #1549494)
    - SAUCE: (noup) KVM: arm/arm64: Handle forward time correction gracefully

  * linux: Enforce signed module loading when UEFI secure boot (LP: #1566221)
    - [Config] CONFIG_EFI_SECURE_BOOT_SIG_ENFORCE=y

  * s390/cpumf: Fix lpp detection (LP: #1555344)
    - s390/facilities: use stfl mnemonic instead of insn magic
    - s390/facilities: always use lowcore's stfle field for storing facility bits
    - s390/cpumf: Fix lpp detection

  * s390x kernel image needs weightwatchers (LP: #1536245)
    - [Config] s390x: Use compressed kernel bzImage

  * Surelock GA2 SP1: surelock02p05: Not seeing sgX devices for LUNs after
    upgrading to Ubuntu 16.04 (LP: #1567581)
    - Revert "UBUNTU: SAUCE: (noup) powerpc/pci: Assign fixed PHB number based on
      device-tree properties"

  * Backport upstream bugfixes to ubuntu-16.04 (LP: #1555765)
    - cpufreq: powernv: Define per_cpu chip pointer to optimize hot-path
    - Revert "cpufreq: postfix policy directory with the first CPU in related_cpus"
    - cpufreq: powernv: Add sysfs attributes to show throttle stats

  * systemd-modules-load.service: Failing due to missing module 'ib_iser' (LP: #1566468)
    - [Config] Add ib_iser to generic inclusion list

  * thunderx nic performance improvements (LP: #1567093)
    - net: thunderx: Set recevie buffer page usage count in bulk
    - net: thunderx: Adjust nicvf structure to reduce cache misses

  * fixes for thunderx nic in multiqueue mode (LP: #1567091)
    - net: thunderx: Fix for multiqset not configured upon interface toggle
    - net: thunderx: Fix for HW TSO not enabled for secondary qsets
    - net: thund...

Read more...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-raspi2 (Ubuntu Xenial):
status: New → Confirmed
Changed in linux-raspi2 (Ubuntu):
status: New → Confirmed
HITMAn (hitmansoft) wrote :

I've noticed the same problem on official ubuntu server 16.04.1 image for raspi2 and marked this bug as also affecting linux-raspi2 package.

Simon Kropf (simon-kropf) wrote :

Same bug here on ubuntu server 16.04 raspi3 image.

Install of pptpd via apt-get failed.
Fixed tempoarily -> ib_iser to #ib_iser in /lib/modules-load.d/open-iscsi.conf

Confirmed on rpi2 image. Removing open-iscsi package fixes the 'problem' for me.

Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1566468

tags: added: iso-testing
Adam Smith (adamsmith) wrote :

I've created a new bug for the raspi2 server image - Bug #1765154

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-kvm (Ubuntu Xenial):
status: New → Confirmed
Changed in linux-kvm (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers