Master SC001-standalone jobs failing consistently with error "Service multipathd has not started yet"

[Pooja Jadhav]

scenario001-standalone jobs failing consistently with error "Service multipathd has not started yet" with below traceback :

2021-10-18 05:57:41.377462 | fa163e2d-137b-0c96-4cde-000000003007 | TASK | Check if /etc/sysconfig/podman_drop_in exists
2021-10-18 05:57:41.599118 | fa163e2d-137b-0c96-4cde-000000003007 | OK | Check if /etc/sysconfig/podman_drop_in exists | standalone
2021-10-18 05:57:41.600356 | fa163e2d-137b-0c96-4cde-000000003007 | TIMING | tripleo_container_manage : Check if /etc/sysconfig/podman_drop_in exists | standalone | 0:25:43.678348 | 0.22s
2021-10-18 05:57:41.620841 | fa163e2d-137b-0c96-4cde-000000003008 | TASK | Set podman_drop_in fact
2021-10-18 05:57:41.648901 | fa163e2d-137b-0c96-4cde-000000003008 | OK | Set podman_drop_in fact | standalone
2021-10-18 05:57:41.650181 | fa163e2d-137b-0c96-4cde-000000003008 | TIMING | tripleo_container_manage : Set podman_drop_in fact | standalone | 0:25:43.728151 | 0.03s
2021-10-18 05:57:41.672870 | fa163e2d-137b-0c96-4cde-000000003009 | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_3
2021-10-18 05:58:14.714582 | fa163e2d-137b-0c96-4cde-000000003009 | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_3 | standalone | error={"changed": false, "msg": "Service multipathd has not started yet"}
2021-10-18 05:58:14.716851 | fa163e2d-137b-0c96-4cde-000000003009 | TIMING | tripleo_container_manage : Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_3 | standalone | 0:26:16.794816 | 33.04s

This issue is facing at integration and component lines as well.

Detailed Logs :

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/a6dd1fb/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-glance-master/f2b496a/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-common-master/b0f3aa2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[Alan Bishop (alan-bishop)]

I think something else is causing this. [1] does indicate a number of failures, but not runs fail and not all the failures are are due to "Service multipathd has not started yet"

Furthermore, [2] shows the multipathd container is running and [3] doesn't show any errors.

[1] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario001-standalone
[2] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/a6dd1fb/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz
[3] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/a6dd1fb/logs/undercloud/var/log/extra/podman/containers/multipathd/stdout.log.txt.gz

[Douglas Viroel (dviroel)]

tripleo_multipathd.service is failing to find multipathd pid here:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/a6dd1fb/logs/undercloud/var/log/extra/failed_services.txt.gz

[Ronelle Landy (rlandy)]

Also hitting check now: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dc2/814394/1/check/tripleo-ci-centos-8-scenario001-standalone/dc25db0/logs/undercloud/home/zuul/standalone_deploy.log

[Ronelle Landy (rlandy)]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario001-standalone

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/814458
Committed: https://opendev.org/openstack/tripleo-ansible/commit/22d2773ddfb1aa23820b4a21982f3f244d3c2a1a
Submitter: "Zuul (22348)"
Branch: master

commit 22d2773ddfb1aa23820b4a21982f3f244d3c2a1a
Author: Ronelle Landy <email address hidden>
Date: Mon Oct 18 18:27:22 2021 +0000

    Revert "Remove execpost and align systemd timeouts"

    This reverts commit fdfc59cd1d1024d132e6046f3ba4746fed23c8c2.

    Reason for revert: https://bugs.launchpad.net/tripleo/+bug/1947548

    Closes-Bug: #1947548
    Change-Id: I7b43abc796f1b3b1db2cebdaa035ab71dab0f9d0

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 4.2.0 release.