systems-module-load.service is not getting started in overcloud deploy

[Amol Kahat]

Error

Unable to start service systemd-modules-load.service

Actual Error
============

2022-01-05 05:44:02 | 2022-01-05 05:43:58.046688 | fa163ecc-7782-df20-9847-000000001650 | FATAL | Modules reload | overcloud-controller-0 | error={"changed": false, "msg": "Unable to start service systemd-modules-load.service: Job for systemd-modules-load.service failed because the control process exited with error code.\nSee \"systemctl status systemd-modules-load.service\" and \"journalctl -xe\" for details.\n"}
2022-01-05 05:44:02 | 2022-01-05 05:43:58.048487 | fa163ecc-7782-df20-9847-000000001650 | TIMING | tripleo-kernel : Modules reload | overcloud-controller-0 | 0:03:21.555599 | 0.55s
2022-01-05 05:44:02 | 2022-01-05 05:43:58.529070 | fa163ecc-7782-df20-9847-000000001650 | FATAL | Modules reload | overcloud-controller-1 | error={"changed": false, "msg": "Unable to start service systemd-modules-load.service: Job for systemd-modules-load.service failed because the control process exited with error code.\nSee \"systemctl status systemd-modules-load.service\" and \"journalctl -xe\" for details.\n"}
2022-01-05 05:44:02 | 2022-01-05 05:43:58.529763 | fa163ecc-7782-df20-9847-000000001650 | TIMING | tripleo-kernel : Modules reload | overcloud-controller-1 | 0:03:22.036865 | 0.54s
2022-01-05 05:44:02 | 2022-01-05 05:43:59.714889 | fa163ecc-7782-df20-9847-00000000164d | CHANGED | Remove dracut-config-generic | overcloud-controller-2
2022-01-05 05:44:02 | 2022-01-05 05:43:59.716297 | fa163ecc-7782-df20-9847-00000000164d | TIMING | tripleo-kernel : Remove dracut-config-generic | overcloud-controller-2 | 0:03:23.223416 | 2.94s
2022-01-05 05:44:02 | 2022-01-05 05:43:59.782717 | fa163ecc-7782-df20-9847-00000000164e | TASK | Ensure the /etc/modules-load.d/ directory exists
2022-01-05 05:44:02 | 2022-01-05 05:44:00.102653 | fa163ecc-7782-df20-9847-00000000164e | OK | Ensure the /etc/modules-load.d/ directory exists | overcloud-controller-2
2022-01-05 05:44:02 | 2022-01-05 05:44:00.104416 | fa163ecc-7782-df20-9847-00000000164e | TIMING | tripleo-kernel : Ensure the /etc/modules-load.d/ directory exists | overcloud-controller-2 | 0:03:23.611530 | 0.32s
2022-01-05 05:44:02 | 2022-01-05 05:44:00.167325 | fa163ecc-7782-df20-9847-00000000164f | TASK | Write list of modules to load at boot
2022-01-05 05:44:02 | 2022-01-05 05:44:01.151096 | fa163ecc-7782-df20-9847-00000000164f | CHANGED | Write list of modules to load at boot | overcloud-controller-2
2022-01-05 05:44:02 | 2022-01-05 05:44:01.153209 | fa163ecc-7782-df20-9847-00000000164f | TIMING | tripleo-kernel : Write list of modules to load at boot | overcloud-controller-2 | 0:03:24.660304 | 0.99s
2022-01-05 05:44:02 | 2022-01-05 05:44:01.231278 | fa163ecc-7782-df20-9847-000000001650 | TASK | Modules reload
2022-01-05 05:44:02 | 2022-01-05 05:44:01.806541 | fa163ecc-7782-df20-9847-000000001650 | FATAL | Modules reload | overcloud-controller-2 | error={"changed": false, "msg": "Unable to start service systemd-modules-load.service: Job for systemd-modules-load.service failed because the control process exited with error code.\nSee \"systemctl status systemd-modules-load.service\" and \"journalctl -xe\" for details.\n"}
2022-01-05 05:44:02 | 2022-01-05 05:44:01.808336 | fa163ecc-7782-df20-9847-000000001650 | TIMING | tripleo-kernel : Modules reload | overcloud-controller-2 | 0:03:25.315448 | 0.58s

Logs
====
- https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-train/7c1b50e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-ussuri/d8015d1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/02ddf5d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/cc26afc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Amol Kahat (amolkahat)]

Service systemd-modules-load.service failed to start

● systemd-modules-load.service - Load Kernel Modules
   Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2022-01-05 14:26:11 UTC; 4min 7s ago
     Docs: man:systemd-modules-load.service(8)
           man:modules-load.d(5)
  Process: 14681 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
 Main PID: 14681 (code=exited, status=1/FAILURE)

Jan 05 14:26:11 overcloud-controller-1 systemd[1]: Starting Load Kernel Modules...
Jan 05 14:26:11 overcloud-controller-1 systemd-modules-load[14681]: Inserted module 'br_netfilter'
Jan 05 14:26:11 overcloud-controller-1 systemd-modules-load[14681]: Module 'msr' is builtin
Jan 05 14:26:11 overcloud-controller-1 systemd-modules-load[14681]: Failed to insert 'ipmi_si': No such device
Jan 05 14:26:11 overcloud-controller-1 systemd[1]: systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
Jan 05 14:26:11 overcloud-controller-1 systemd[1]: systemd-modules-load.service: Failed with result 'exit-code'.
Jan 05 14:26:11 overcloud-controller-1 systemd[1]: Failed to start Load Kernel Modules.

- https://logserver.rdoproject.org/50/37350/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/9e0896f/logs/overcloud-novacompute-0/var/log/extra/services.txt.gz
- https://logserver.rdoproject.org/50/37350/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-victoria/ed39f66/logs/overcloud-novacompute-0/var/log/extra/services.txt.gz
- https://logserver.rdoproject.org/50/37350/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/b6aee90/logs/overcloud-controller-1/var/log/extra/services.txt.gz

[Ronelle Landy (rlandy)]

Possibly related BZ:

https://bugzilla.redhat.com/show_bug.cgi?id=1702452

(c8 is still 4.18 , and that bz was from 5.1, so I suspect same bug, just was backported into stream ?)

http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/?C=M;O=D

has kernel update on 12/22 and other updates on 01/03

with latest failure: latest failure log: https://logserver.rdoproject.org/01/822001/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/dd33688/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Ronelle Landy (rlandy)]

stevebaker[m]> rlandy|ruck: Getting a minimal reproducer would help to know what bug to raise. You could start by sourcing the centos8 qcow2 which was used to build this overcloud and see if that boots in this cloud. If that works try using the same diskimage-builder to build a basic centos vm image and boot that. Hopefully it is not necessary to build a centos partition image to reproduce

[Ronelle Landy (rlandy)]

https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/hardware.py#L138

[Ronelle Landy (rlandy)]

some investigation into the differences of the packages in the image build:

last passing:

2022-01-04 18:10:16.773 | kexec-tools x86_64 2.0.20-63.el8 baseos 521 k

first failing:

 kexec-tools x86_64 2.0.20-67.el8 baseos 522 k

[Ronelle Landy (rlandy)]

Look at fs002, (which builds images) you can see the start of the failure on 01/05:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master

[Ronelle Landy (rlandy)]

Similar BZ open:

Failed to insert 'ipmi_si': No such device causes systemd-modules-load.service failed - logged against RHEL 8.6

https://bugzilla.redhat.com/show_bug.cgi?id=2030993

[chandan kumar (chkumar246)]

Testing here with the latest centos stream 8 image: https://review.rdoproject.org/r/c/testproject/+/37791

[Ronelle Landy (rlandy)]

Note from ykarel about cause:

https://git.centos.org/rpms/fwupd/c/6c27b54f9ba05f8860ee1fd8d70450b1d549a68a?branch=c8s

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/823694

[Amol Kahat (amolkahat)]

Review: https://review.opendev.org/c/openstack/tripleo-ci/+/823694
Testproject: https://review.rdoproject.org/r/c/testproject/+/36487

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/823737

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/823738

[Alfredo Moralejo (amoralej)]

Another workaround would be to create /etc/modprobe.d/ipmi_si.conf with following content during image creation:

blacklist ipmi_si

That will prevent from trying to enable the module,

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/823798

[chandan kumar (chkumar246)]

823798: Blacklist ipmi_si for CentOS 8 distros | https://review.opendev.org/c/openstack/tripleo-ansible/+/823798 and testing here with testproject: https://review.rdoproject.org/r/c/testproject/+/37857

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/823736
Committed: https://opendev.org/openstack/tripleo-ci/commit/018f3ecf7e2100d873391a354086f17469514ecf
Submitter: "Zuul (22348)"
Branch: master

commit 018f3ecf7e2100d873391a354086f17469514ecf
Author: Ronelle Landy <email address hidden>
Date: Thu Jan 6 18:16:25 2022 -0500

    Remove fwupd-redfish.conf file from overcloud images

    Right now, the fwupd package is attempting to load the module ipmi_si
    and this one is failing to be load. This patch customize the qcow2 image
    removing the fwupd-redfish.conf that is causing this problem.
    It is, right now hard to just exclude the package because of the
    dependences, as well as downgrade, because of how the repositories are
    configured in the image, so the best approach to temporarly solve the
    problem is in this way with virt-customize.

    Related-Bug: #1956441

    Signed-off-by: Amol Kahat <email address hidden>

    Change-Id: I07b2344f85c6997c10c3e50a86ba8e16dfcb1355

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/823738

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/823737

[OpenStack Infra (hudson-openstack)]

Change abandoned by "amolkahat <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/823694
Reason: https://review.opendev.org/c/openstack/tripleo-ci/+/823736

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/823798
Reason: 823736: Remove fwupd-redfish.conf file from overcloud images | https://review.opendev.org/c/openstack/tripleo-ci/+/823736

[Alan Pevec (apevec)]

https://bugzilla.redhat.com/show_bug.cgi?id=2030993 was fixed in RHEL 8.6