periodic-tripleo-rhel-8-buildimage-overcloud-full-master is failing - failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory

[Ronelle Landy]

periodic-tripleo-rhel-8-buildimage-overcloud-full-master is failing - with the following errors:

2019-11-04 12:29:25.214 | + systemctl enable network
2019-11-04 12:29:25.219 | network.service is not a native service, redirecting to systemd-sysv-install.
2019-11-04 12:29:25.219 | Executing: /usr/lib/systemd/systemd-sysv-install enable network
2019-11-04 12:29:25.221 | failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory
2019-11-04 12:29:25.891 | Unmount /tmp/dib_build.yyNNCM8h/mnt/tmp/yum
2019-11-04 12:29:25.925 | Unmount /tmp/dib_build.yyNNCM8h/mnt/tmp/in_target.d
2019-11-04 12:29:25.948 | Unmount /tmp/dib_build.yyNNCM8h/mnt/sys
2019-11-04 12:29:25.975 | Unmount /tmp/dib_build.yyNNCM8h/mnt/proc
2019-11-04 12:29:26.001 | Unmount /tmp/dib_build.yyNNCM8h/mnt/dev/pts
2019-11-04 12:29:26.025 | Unmount /tmp/dib_build.yyNNCM8h/mnt/dev
2019-11-04 12:29:26.481 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here

The full log is linked below:

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/4c6d8f7/overcloud-full.log

The RHEL-8 images directory shows the missing overcloud images:

[root@rcn-share rdo_trunk]# ls -la ad27bcc3ea43100cb970e3d28eb68099479f1a37_6e3b098e
total 473668
drwxrwxr-x. 2 centos centos 72 Nov 4 18:30 .
drwxr-xr-x. 90 centos centos 8192 Nov 4 18:30 ..
-rw-rw-r--. 1 centos centos 485017600 Nov 4 18:30 ironic-python-agent.tar
-rw-rw-r--. 1 centos centos 58 Nov 4 18:30 ironic-python-agent.tar.md5

This missing image is causing tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001 to fail:

2019-11-04 10:46:29.142187 | primary | fatal: [undercloud]: FAILED! => {
2019-11-04 10:46:29.142224 | primary | "attempts": 10,
2019-11-04 10:46:29.142231 | primary | "changed": true,
2019-11-04 10:46:29.142236 | primary | "cmd": [
2019-11-04 10:46:29.142241 | primary | "curl",
2019-11-04 10:46:29.142245 | primary | "-skfL",
2019-11-04 10:46:29.142250 | primary | "http://38.145.34.141/rcm-guest/images/redhat8/master
/rdo_trunk/tripleo-ci-testing/overcloud-full.tar.md5"
2019-11-04 10:46:29.142255 | primary | ],

http://logs.rdoproject.org/00/692600/15/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/4bcc135/job-output.txt

[Ronelle Landy (rlandy)]

May have been just a node hitch ...

http://logs.rdoproject.org/42/23542/3/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/88d6426/ worked

[Ronelle Landy (rlandy)]

Occurs sporadically - happened again today

[Ronelle Landy (rlandy)]

Look somewhat similar to https://bugs.launchpad.net/tripleo/+bug/1846391

[Luke Short (ekultails)]

I think this might actually be a problem in puppet-tripleo (or elsewhere). There is no mention of the "tripleo-bootstrap" role being executed and the playbook would be outputting task information such as "Ensure network service is enabled". It looks like a Bash script is written to a temporary file that is hardcoded to enable the network service.

```
2019-11-04 18:28:17.214 | dib-run-parts 51-disable-libvirtd-service completed
2019-11-04 18:28:17.214 | dib-run-parts Running /tmp/in_target.d/post-install.d/51-enable-network-service
2019-11-04 18:28:17.217 | + set -o pipefail
2019-11-04 18:28:17.217 | + systemctl enable network
2019-11-04 18:28:17.222 | network.service is not a native service, redirecting to systemd-sysv-install.
2019-11-04 18:28:17.222 | Executing: /usr/lib/systemd/systemd-sysv-install enable network
2019-11-04 18:28:17.224 | failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory
```

[Ronelle Landy (rlandy)]

https://github.com/openstack/tripleo-puppet-elements/blob/master/elements/overcloud-base/post-install.d/51-enable-network-service

[Ronelle Landy (rlandy)]

Recording come comments from irc discussion:

<ianw> i don't think dib has changed any of this recently
<mwhahaha> yea i don't think so either, it's failing in the systemd wrapper bits but that hasn't changed according to the rpms that are getting isntalled
<mwhahaha> the systemd-sysv-install comes from chkconfig anyway
<mwhahaha> only suggestion i'd have is to run it with debugging enabled
<ianw> add "-x", or you can set DIB_DEBUG_TRACE=1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693190

[Ronelle Landy (rlandy)]

The patch fails with:

2019-11-06 15:24:49.873 | + set -o pipefail
2019-11-06 15:24:49.873 | + [[ ! -f /usr/lib/systemd/system/network.service ]]
2019-11-06 15:24:49.873 | + systemctl --now enable NetworkManager
2019-11-06 15:24:49.878 | System has not been booted with systemd as init system (PID 1). Can't operate.
2019-11-06 15:24:49.878 | Failed to connect to bus: Host is down

[Luke Short (ekultails)]

That error indicates that this is probably running inside of a container which, by default, does not have the bind mounts and access to systemd resources. The unpatched version of this script also calls systemctl to manage the network service so in theory it should already be failing with that same message.

I don't know why or how this is running inside of a container. It should only be executed directly on the Overcloud hosts.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693370

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693370
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=63859da17b4f4e25b4817950544a5daf50270e8a
Submitter: Zuul
Branch: master

commit 63859da17b4f4e25b4817950544a5daf50270e8a
Author: Alex Schultz <email address hidden>
Date: Thu Nov 7 08:18:34 2019 -0700

    Use chkconfig

    The network.service is a legacy config. Use chkconfig directly instead
    of the systemctl version.

    Depends-On: https://review.opendev.org/#/c/693379/
    Change-Id: I1614da55b3e76ef8cca715cd3e1897bc76a2a3fb
    Closes-Bug: #1851274

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/693446

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/693447

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693447
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=78f7e7f6d926b27cd4025fa7c28a7d84d01813f2
Submitter: Zuul
Branch: stable/stein

commit 78f7e7f6d926b27cd4025fa7c28a7d84d01813f2
Author: Alex Schultz <email address hidden>
Date: Thu Nov 7 08:18:34 2019 -0700

    Use chkconfig

    The network.service is a legacy config. Use chkconfig directly instead
    of the systemctl version.

    Depends-On: https://review.opendev.org/#/c/693379/
    Change-Id: I1614da55b3e76ef8cca715cd3e1897bc76a2a3fb
    Closes-Bug: #1851274
    (cherry picked from commit 63859da17b4f4e25b4817950544a5daf50270e8a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693446
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=2ad3189cf86a484cc502fc37797e83ab0e70868a
Submitter: Zuul
Branch: stable/train

commit 2ad3189cf86a484cc502fc37797e83ab0e70868a
Author: Alex Schultz <email address hidden>
Date: Thu Nov 7 08:18:34 2019 -0700

    Use chkconfig

    The network.service is a legacy config. Use chkconfig directly instead
    of the systemctl version.

    Depends-On: https://review.opendev.org/#/c/693379/
    Change-Id: I1614da55b3e76ef8cca715cd3e1897bc76a2a3fb
    Closes-Bug: #1851274
    (cherry picked from commit 63859da17b4f4e25b4817950544a5daf50270e8a)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Luke Short (<email address hidden>) on branch: master
Review: https://review.opendev.org/693190
Reason: Superseded by a simpler patch that uses chkconfig instead: https://review.opendev.org/#/c/693370/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-puppet-elements 12.0.0 release.

[Marios Andreou (marios-b)]

not sure this was fixed see comment #5 @ https://bugs.launchpad.net/tripleo/+bug/1853028 we are still seeing the 2019-11-26 01:55:15.889 | failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory with chkonfig

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-puppet-elements 10.3.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-puppet-elements 11.2.1 release.

[Sagi (Sergey) Shnaidman (sshnaidm)]

Was solved by https://review.rdoproject.org/r/#/c/28041 in third-party CI and was ported to upstream: https://review.opendev.org/#/c/738469/