[master] Job in gate failed with Could not find the requested service tripleo_keystone.service: host

[yatin]

Start or restart systemd services task failed with below error:-

TASK [tripleo_container_manage : Start or restart systemd services] ************
Friday 17 April 2020 12:03:23 +0000 (0:00:02.559) 0:23:41.096 **********
failed: [undercloud] (item=keystone) => changed=false
  ansible_loop_var: container_sysd_name
  container_sysd_name: keystone
  msg: 'Could not find the requested service tripleo_keystone.service: host'
changed: [undercloud] => (item=iscsid)

Logs:- https://a159bad83b0083ae3f18-b71c90f67122d6f7b38b8442bc3a114a.ssl.cf1.rackcdn.com/713439/13/gate/tripleo-ci-centos-8-containers-undercloud-minion/b781b59/logs/undercloud/home/zuul/undercloud_install.log

As per log tripleo_keystone.service exists https://a159bad83b0083ae3f18-b71c90f67122d6f7b38b8442bc3a114a.ssl.cf1.rackcdn.com/713439/13/gate/tripleo-ci-centos-8-containers-undercloud-minion/b781b59/logs/undercloud/etc/systemd/system/tripleo_keystone.service

As per logs daemon reload also executed in a task above:-

TASK [tripleo_container_manage : Force systemd daemon reload if a systemd file changed] ***
Friday 17 April 2020 12:03:19 +0000 (0:00:00.462) 0:23:37.685 **********
ok: [undercloud]

May be some race or corner case, need to find the cause.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/720707

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720707
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=6e71ea60b659485e08d02cff7a9966c14c36a88b
Submitter: Zuul
Branch: master

commit 6e71ea60b659485e08d02cff7a9966c14c36a88b
Author: Emilien Macchi <email address hidden>
Date: Fri Apr 17 09:06:36 2020 -0400

    tripleo_container_manage/systemd: add retries mechanism

    It's possible that in CI environment or in slow servers the systemd
    reload takes time to read new systemd services and reload systemd while
    Ansible already moved on and tries to enable these services.

    Adding a retry mechanism so we retry 5 times with a delay of 5 seconds
    between, until systemd service returns success as a result.

    Change-Id: I2bb39d2aae68a0eadaad92cf0fc4e3506fbfc9ca
    Closes-Bug: #1873453

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/721332

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721332
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=4e903e2fcf524e3baee73974cae333275cb2c4e9
Submitter: Zuul
Branch: stable/train

commit 4e903e2fcf524e3baee73974cae333275cb2c4e9
Author: Emilien Macchi <email address hidden>
Date: Fri Apr 17 09:06:36 2020 -0400

    tripleo_container_manage/systemd: add retries mechanism

    It's possible that in CI environment or in slow servers the systemd
    reload takes time to read new systemd services and reload systemd while
    Ansible already moved on and tries to enable these services.

    Adding a retry mechanism so we retry 5 times with a delay of 5 seconds
    between, until systemd service returns success as a result.

    Change-Id: I2bb39d2aae68a0eadaad92cf0fc4e3506fbfc9ca
    Closes-Bug: #1873453
    (cherry picked from commit 6e71ea60b659485e08d02cff7a9966c14c36a88b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 0.6.0 release.