rhel 8 tripleo Destination directory /etc/modules-load.d does not exist

[wes hayutin]

2019-10-07 10:50:36 | TASK [tripleo-kernel : Write list of modules to load at boot] ******************
2019-10-07 10:50:36 | fatal: [overcloud-controller-0]: FAILED! => {
2019-10-07 10:50:36 | "changed": false,
2019-10-07 10:50:36 | "checksum": "aa9e98ce9e5357ff5b188b81ceac53340b7c8bfc"
2019-10-07 10:50:36 | }
2019-10-07 10:50:36 |
2019-10-07 10:50:36 | MSG:
2019-10-07 10:50:36 |
2019-10-07 10:50:36 | Destination directory /etc/modules-load.d does not exist
2019-10-07 10:50:36 | fatal: [overcloud-controller-1]: FAILED! => {
2019-10-07 10:50:36 | "changed": false,
2019-10-07 10:50:36 | "checksum": "aa9e98ce9e5357ff5b188b81ceac53340b7c8bfc"
2019-10-07 10:50:36 | }
2019-10-07 10:50:36 |
2019-10-07 10:50:36 | MSG:
2019-10-07 10:50:36 |
2019-10-07 10:50:36 | Destination directory /etc/modules-load.d does not exist
2019-10-07 10:50:36 | fatal: [overcloud-controller-2]: FAILED! => {
2019-10-07 10:50:36 | "changed": false,
2019-10-07 10:50:36 | "checksum": "aa9e98ce9e5357ff5b188b81ceac53340b7c8bfc"
2019-10-07 10:50:36 | }

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/5546dbc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[wes hayutin (weshayutin)]

https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo-kernel/tasks/main.yml#L48-L53

    - name: Write list of modules to load at boot
      template:
        src: "tripleo-modprobe.conf.j2"
        dest: "/etc/modules-load.d/99-tripleo.conf"
      notify:
        - Modules reload

[wes hayutin (weshayutin)]

That directory does not exist on nodepool rhel8 nodes, it does on my local rhel8 install though.

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/5546dbc/logs/overcloud-controller-0/etc/

[wes hayutin (weshayutin)]

hrm.. afaict.. systemd-udev-239-13.el8_0.5.x86_64 should provide that dir.

[root@rheltestbox ~]# rpm -q --whatprovides /etc/modules-load.d/
systemd-udev-239-13.el8_0.5.x86_64

It is installed on the node:
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/5546dbc/logs/overcloud-controller-0/var/log/extra/rpm-list.txt.gz

[wes hayutin (weshayutin)]

ah.. root cause..
  systemd-modules-load.service loaded inactive dead Load Kernel Modules

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/5546dbc/logs/overcloud-controller-0/var/log/extra/services.txt.gz

[wes hayutin (weshayutin)]

This is most likely a selinux issue.. as I see it's enforcing.

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/5546dbc/logs/overcloud-controller-0/etc/selinux/

[wes hayutin (weshayutin)]

https://review.opendev.org/#/c/687345/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/687361

[wes hayutin (weshayutin)]

hrm.. getting selinux set to permissive did NOT resolve the issue.

http://logs.rdoproject.org/61/687361/1/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/3c37c69/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

http://logs.rdoproject.org/61/687361/1/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/3c37c69/logs/overcloud-controller-0/etc/selinux/

[wes hayutin (weshayutin)]

maybe this is killing it?
https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo-module-load/tasks/main.yml#L56

[wes hayutin (weshayutin)]

2019-10-08 20:06:23 |
2019-10-08 20:06:23 | TASK [tripleo-kernel : Write list of modules to load at boot] ******************
2019-10-08 20:06:23 | changed: [overcloud-novacompute-0] => {
2019-10-08 20:06:23 | "changed": true,
2019-10-08 20:06:23 | "checksum": "aa9e98ce9e5357ff5b188b81ceac53340b7c8bfc",
2019-10-08 20:06:23 | "dest": "/etc/modules-load.d/99-tripleo.conf",
2019-10-08 20:06:23 | "gid": 0,
2019-10-08 20:06:23 | "group": "root",
2019-10-08 20:06:23 | "md5sum": "26aebdd43b2fd9c998bd7ce29ae7a658",
2019-10-08 20:06:23 | "mode": "0644",
2019-10-08 20:06:23 | "owner": "root",
2019-10-08 20:06:23 | "secontext": "system_u:object_r:etc_t:s0",
2019-10-08 20:06:23 | "size": 32,
2019-10-08 20:06:23 | "src": "/home/tripleo-admin/.ansible/tmp/ansible-tmp-1570579565.346318-2746805072109/source",
2019-10-08 20:06:23 | "state": "file",
2019-10-08 20:06:23 | "uid": 0
2019-10-08 20:06:23 | }

http://logs.rdoproject.org/19/687419/4/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/b7138c0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687361
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=ae400a009f8bb16adf8c4714c6dd9543e32d5d39
Submitter: Zuul
Branch: master

commit ae400a009f8bb16adf8c4714c6dd9543e32d5d39
Author: Wes Hayutin <email address hidden>
Date: Tue Oct 8 11:15:55 2019 -0600

    RHEL 8 upstream needs to be permissive

    Related-Bug: #1847282
    Change-Id: If2148ed57b39b6a70b88c4d1585ea3ae37057bcc

[wes hayutin (weshayutin)]

this is fixed by https://review.opendev.org/#/c/687419/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687419
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=0bcb4b906e7204f94193b65e5f55550a7a21475f
Submitter: Zuul
Branch: master

commit 0bcb4b906e7204f94193b65e5f55550a7a21475f
Author: Kevin Carter <email address hidden>
Date: Tue Oct 8 16:00:39 2019 -0500

    Tighten up how we add and remove kernel modules

    This change adjusts our two roles that configure kernel modules.
    These two roles serve slightly different purposes so they both
    must exist at this time. The change being made will ensure that
    the system module-load service is in reloaded anytime modules
    change, and that the `/etc/modules-load.d` directory exists
    before assuming we can write new configuration files there.

    > Molecule configuration for the tripleo-module-load role has
      been updated to ensure the role is functionally testing the
      code paths with real system values.

    Closes-Bug: #1847282
    Change-Id: Id20d7415e8430c66f3afc8f86e0e813a54ee3ca2
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 1.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/747632

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/747632