[master] scenario 001/002 Deployments failing with:- Failed to add dependency: Unit file tripleo_ceilometer_gnocchi_upgrade.service does not exist.

[yatin]

Scenario001/002[1] Deployments in Ussuri are failing with below Error:-
TASK [tripleo_container_manage : Add systemd requires for healthchecks] ********
Thursday 16 April 2020 04:20:13 +0000 (0:00:01.292) 1:02:50.275 ********
failed: [standalone] (item=ceilometer_gnocchi_upgrade) => changed=true
  ansible_loop_var: container_requires_timer_name
  cmd:
  - systemctl
  - add-requires
  - tripleo_ceilometer_gnocchi_upgrade.service
  - tripleo_ceilometer_gnocchi_upgrade_healthcheck.timer
  container_requires_timer_name: ceilometer_gnocchi_upgrade
  delta: '0:00:00.012157'
  end: '2020-04-16 04:20:13.753857'
  msg: non-zero return code
  rc: 1
  start: '2020-04-16 04:20:13.741700'
  stderr: 'Failed to add dependency: Unit file tripleo_ceilometer_gnocchi_upgrade.service does not exist.'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

Example logs:-
- https://980825aa441338566c98-f5b371aed546eed3ac03c77e9adc0c03.ssl.cf2.rackcdn.com/713439/13/check/tripleo-ci-centos-8-scenario001-standalone/a5ae04f/logs/undercloud/home/zuul/standalone_deploy.log
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_00b/713439/13/check/tripleo-ci-centos-8-scenario002-standalone/00b3755/logs/undercloud/home/zuul/standalone_deploy.log

Likely caused by https://review.opendev.org/#/c/718545/.

[1] http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario001-standalone&job_name=tripleo-ci-centos-8-scenario002-standalone

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/720451

[Emilien Macchi (emilienm)]

I'll take the bug. In the meantime, let's revert: https://review.opendev.org/#/c/720451

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/720453

[Emilien Macchi (emilienm)]

1) https://review.opendev.org/720453 will cleanup the wrongly configured container in THT: https://review.opendev.org/720453

2) https://review.opendev.org/720462 will add a safeguard so we don't hit that problem anymore in the future and also some testing to test that wrong scenario.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/720580

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.opendev.org/720451

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720462
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=04f16051cc112180d4fb66fc002a453c1ede23d2
Submitter: Zuul
Branch: master

commit 04f16051cc112180d4fb66fc002a453c1ede23d2
Author: Emilien Macchi <email address hidden>
Date: Thu Apr 16 08:51:32 2020 -0400

    tripleo_container_manage: add safeguard against wrong healthcheck config

    If a container config has by mistake a healthcheck but no systemd
    restart policy, we don't want to manage the healthcheck because it
    requires its service to be created.

    To prevent that situation, we'll create the healthchecks only if they
    are already part of the systemd services list that was created earlier.
    For that, we're using the intersect() filter which allows to
    get the intersection of 2 lists (systemd services and healthchecks to
    create).

    Adding molecule coverage to test this scenario.

    Closes-Bug: #1873249
    Change-Id: Id5cc784bae597def0648f07d28b6463b387d2212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720580
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=40669aeee3c538c8045d7a9d07364bba28ececba
Submitter: Zuul
Branch: stable/train

commit 40669aeee3c538c8045d7a9d07364bba28ececba
Author: Emilien Macchi <email address hidden>
Date: Wed Apr 8 16:10:23 2020 -0400

    [train-squash] tripleo_container_manage: backport systemd optimizations

    This is a combination of 2 commits to avoid regressions.
    (see https://bugs.launchpad.net/tripleo/+bug/1873249)

    This is the 1st commit message:

    tripleo_container_manage: optimize systemd services/healthcheck bootstrap

    Separate the creation of systemd files & service restarts so we don't
    call systemd too many times and makes the deployment faster.

    It also uses a new filter that will read register data to figure out
    what systemd files changed so what containers need a restart.

    Change-Id: I16596a5b262642a678a8b8b123384fc387f69c70
    (cherry picked from commit 761e5cbdd589f582003f4e2f27a03c313de8a852)

    This is the commit message #2:

    tripleo_container_manage: add safeguard against wrong healthcheck config

    If a container config has by mistake a healthcheck but no systemd
    restart policy, we don't want to manage the healthcheck because it
    requires its service to be created.

    To prevent that situation, we'll create the healthchecks only if they
    are already part of the systemd services list that was created earlier.
    For that, we're using the intersect() filter which allows to
    get the intersection of 2 lists (systemd services and healthchecks to
    create).

    Adding molecule coverage to test this scenario.

    Closes-Bug: #1873249
    Change-Id: Id5cc784bae597def0648f07d28b6463b387d2212
    (cherry picked from commit 04f16051cc112180d4fb66fc002a453c1ede23d2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720453
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d78ce138f67aee2f2bf9c5d0db89a5b1e4b16fb0
Submitter: Zuul
Branch: master

commit d78ce138f67aee2f2bf9c5d0db89a5b1e4b16fb0
Author: Emilien Macchi <email address hidden>
Date: Thu Apr 16 08:24:32 2020 -0400

    Remove healthcheck from ceilometer_gnocchi_upgrade container

    Not sure why ceilometer_gnocchi_upgrade has healthcheck key in its
    configuration but the container isn't managed by systemd and it's only
    running a one-off shell command to upgrade ceilometer so it should not
    have a healthcheck.

    Change-Id: I38aa38099a2d759c1b1f2e14d64c36d52c0b1c0e
    Related-Bug: #1873249

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/723910

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/723910
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7d2aac5264071bcca807fbd02dd26ae01aaeb294
Submitter: Zuul
Branch: stable/train

commit 7d2aac5264071bcca807fbd02dd26ae01aaeb294
Author: Emilien Macchi <email address hidden>
Date: Thu Apr 16 08:24:32 2020 -0400

    Remove healthcheck from ceilometer_gnocchi_upgrade container

    Not sure why ceilometer_gnocchi_upgrade has healthcheck key in its
    configuration but the container isn't managed by systemd and it's only
    running a one-off shell command to upgrade ceilometer so it should not
    have a healthcheck.

    Change-Id: I38aa38099a2d759c1b1f2e14d64c36d52c0b1c0e
    Related-Bug: #1873249
    (cherry picked from commit d78ce138f67aee2f2bf9c5d0db89a5b1e4b16fb0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 0.6.0 release.