No key/value pairs provided, at least one is required for this action to succeed

[Ananya Banerjee]

periodic-tripleo-ci-centos-9-scenario004-standalone-master fails with

2021-10-27 23:33:26,211 p=50404 u=root n=ansible | 2021-10-27 23:33:26.210692 | 0e2f980e-4b51-4ae4-b727-669c1edba66b | INCLUDED | /home/zuul-worker/tripleo-deploy/standalone-ansible-zrqgu7ka/external_deploy_steps_tasks_step1.yaml | undercloud
2021-10-27 23:33:26,228 p=50404 u=root n=ansible | 2021-10-27 23:33:26.228423 | fa163edf-f36f-cd20-1323-000000001226 | TASK | Set some tripleo-ansible facts
2021-10-27 23:33:26,254 p=50404 u=root n=ansible | 2021-10-27 23:33:26.254375 | fa163edf-f36f-cd20-1323-000000001226 | OK | Set some tripleo-ansible facts | undercloud
2021-10-27 23:33:26,269 p=50404 u=root n=ansible | 2021-10-27 23:33:26.269503 | fa163edf-f36f-cd20-1323-000000001228 | TASK | set tripleo-ansible group vars
2021-10-27 23:33:26,286 p=50404 u=root n=ansible | 2021-10-27 23:33:26.285460 | fa163edf-f36f-cd20-1323-000000001228 | OK | set tripleo-ansible group vars | undercloud
2021-10-27 23:33:26,301 p=50404 u=root n=ansible | 2021-10-27 23:33:26.301129 | fa163edf-f36f-cd20-1323-00000000122a | TASK | set tripleo-ansible ceph dashboard vars
2021-10-27 23:33:26,327 p=50404 u=root n=ansible | 2021-10-27 23:33:26.327083 | fa163edf-f36f-cd20-1323-00000000122a | FATAL | set tripleo-ansible ceph dashboard vars | undercloud | error={"changed": false, "msg": "No key/value pairs provided, at least one is required for this action to succeed"}

https://logserver.rdoproject.org/65/36465/1/check/periodic-tripleo-ci-centos-9-scenario004-standalone-master/1aadd0d/logs/undercloud/home/zuul-worker/ansible.log

[Cédric Jeanneret (cjeanner)]

comes from:
deployment/cephadm/ceph-mgr.yaml

This file wasn't change for the past months apparently. So it must come from some higher level in t-h-t...

The block:
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/cephadm/ceph-mgr.yaml#L122-L134

[Francesco Pantano (fmount)]

Hi Cedric,
that's related to the change in ansible 2.11.x vs 2.10.y.
Here the proposed patch [1]

[1] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815782

[Francesco Pantano (fmount)]

testproject's scenario004 fails with the following [1]:

```
2021-10-28 12:11:59,395 p=87916 u=root n=ansible | 2021-10-28 12:11:59.394641 | fa163edd-857b-ec85-78bd-000000000419 | TASK | reload dbus configuration
2021-10-28 12:11:59,696 p=87916 u=root n=ansible | 2021-10-28 12:11:59.694970 | fa163edd-857b-ec85-78bd-000000000419 | FATAL | reload dbus configuration | standalone -> 192.168.24.1 | error={"changed": true, "cmd": ["killall", "-SIGHUP", "dbus-daemon"], "delta": "0:00:00.081489", "end": "2021-10-28 12:11:59.670250", "msg": "non-zero return code", "rc": 1, "start": "2021-10-28 12:11:59.588761", "stderr": "dbus-daemon: no process found", "stderr_lines": ["dbus-daemon: no process found"], "stdout": "", "stdout_lines": []}
2021-10-28 12:11:59,697 p=87916 u=root n=ansible | [WARNING]: ('standalone -> 192.168.24.1',
'fa163edd-857b-ec85-78bd-000000000419') missing from stats
```

and this happens because the dbus-daemon isn't there, so it's expected (and the right thing to do) to see the deployment failing.
Manila uses dbus to interact with the ganesha daemon provided by the ceph-nfs container, however, if the daemon isn't there, Manila can't be properly configured and it fails.
The daemon is present in c8s (Tripleo scenario004), which is green, but not in the rdo scenario004 job provided by testproject [3].

[1] https://logserver.rdoproject.org/80/36480/3/check/periodic-tripleo-ci-centos-9-scenario004-standalone-master/6fb2dd6/logs/undercloud/home/zuul-worker/tripleo-deploy/standalone-ansible-b52ghfef/cephadm/cephadm_command.log

[2] https://2b02ff9ef56057fe875b-5dd750a0358fa670bf85ccc2ef690f1c.ssl.cf5.rackcdn.com/815782/2/check/tripleo-ci-centos-8-scenario004-standalone/6476779/logs/undercloud/var/log/extra/journal.txt

[3] https://logserver.rdoproject.org/80/36480/3/check/periodic-tripleo-ci-centos-9-scenario004-standalone-master/6fb2dd6/logs/undercloud/var/log/extra/journal.txt

[chandan kumar (chkumar246)]

Need dbus-daemon on the host so that https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_cephadm/templates/ceph-nfs.service.j2#L27 can use it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/815863

[chandan kumar (chkumar246)]

chandankumar | fmount: \o regarding this bug https://bugs.launchpad.net/tripleo/+bug/1949033
chandankumar | fmount: how can we get ganesha daemon for cs9?
      fmount | ganesha is provided by the ceph container, so we don't have to get the daemon for cs9
      fmount | chandankumar: dbus-daemon is used by manila to interact (reload) ganesha
      fmount | * w/ ganesha
      fmount | dbus-daemon is missing I guess
      fmount | and the failure is expected
pendevreview | Merged openstack/tripleo-quickstart-extras master: In modify-image don't convert
             | qcow2<->raw https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/809487
         <-- | slaweq (~slaweqmat@2001:470:1af1:101::84a9) has left #tripleo
chandankumar | fmount: if we add dbus-daemon pkg in the manila container does it will fix the job?
     fultonj | chandankumar: that sounds like a good start
          -- | ysandeep|mtg is now known as ysandeep
chandankumar | fultonj: which manila service we need to add that?
       fmount | actually this is needed in the host
       fmount | and it's used by the manila container because it's mounted in the ceph-nfs container
          --> | slaweq (~slaweqmat@2001:470:1af1:101::84a9) has joined #tripleo
       fmount | https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ce
              | phadm/templates/ceph-nfs.service.j2#L27
       fmount | chandankumar: ^^
          --> | frenzy_friday (~anbanerj@103.27.2.30) has joined #tripleo
           -- | amoralej is now known as amoralej|off
 chandankumar | fmount: ah got it
 chandankumar | proposing a patch
       fmount | +1 thank you! :D
           -- | ysandeep is now known as ysandeep|out
       fmount | chandankumar: from the https://bugs.launchpad.net/tripleo/+bug/1949033 you can see the
              | two journal logs and compare what's missing in the c9s host
      fultonj | Thanks Chandan for proposing a patch to get dbus on the container host.
          <-- | lucasagomes (~lucasagom@89.100.20.18) has quit (Quit: Le...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/815948

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/815863
Reason: in favor of https://review.opendev.org/c/openstack/tripleo-ansible/+/815948

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/815948
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b151cd7b139768a17daa22cf48afdd5e9fbf8481
Submitter: "Zuul (22348)"
Branch: master

commit b151cd7b139768a17daa22cf48afdd5e9fbf8481
Author: Francesco Pantano <email address hidden>
Date: Fri Oct 29 11:35:25 2021 +0200

    Reload dbus config should work for both c8 and c9

    c9 ship dbus-broker by default instead of dbus-daemon.
    This means we should make sure the command is valid for
    both the use cases.

    Related-Bug: #1949033
    Change-Id: Ibdf87463d4a86dd225a1f1b89326407628facb65

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/815887

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815782
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/b3863568ad75a3845389677b08a0d756e3c6e0b5
Submitter: "Zuul (22348)"
Branch: master

commit b3863568ad75a3845389677b08a0d756e3c6e0b5
Author: Francesco Pantano <email address hidden>
Date: Thu Oct 28 09:28:33 2021 +0200

    Fix set_fact on the CephMgrAnsibleVars

    The purpose of this patch is react on the issue generated by [1],
    where an empty dict sect_fact is not allowed for ansible >= 2.11.
    For this reason, we're going to always set those variables and
    defer to the ansible layer the evaluation of the DashboardEnabled
    condition.

    [1] https://github.com/ansible/ansible/pull/73684

    Closes-Bug: #1949033
    Change-Id: I4ab2d20df22bdc9b140019f90d1370ef17f09294

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/815887
Committed: https://opendev.org/openstack/tripleo-ansible/commit/d4e7df03387e276547821b41d9a366892fb2bd49
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d4e7df03387e276547821b41d9a366892fb2bd49
Author: Francesco Pantano <email address hidden>
Date: Fri Oct 29 11:35:25 2021 +0200

    Reload dbus config should work for both c8 and c9

    c9 ship dbus-broker by default instead of dbus-daemon.
    This means we should make sure the command is valid for
    both the use cases.

    Related-Bug: #1949033
    Change-Id: Ibdf87463d4a86dd225a1f1b89326407628facb65
    (cherry picked from commit b151cd7b139768a17daa22cf48afdd5e9fbf8481)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/816244

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/816244
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/3e330da2c5d6f4f07b402e358adb3cbdbb504d79
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3e330da2c5d6f4f07b402e358adb3cbdbb504d79
Author: Francesco Pantano <email address hidden>
Date: Thu Oct 28 09:28:33 2021 +0200

    Fix set_fact on the CephMgrAnsibleVars

    The purpose of this patch is react on the issue generated by [1],
    where an empty dict sect_fact is not allowed for ansible >= 2.11.
    For this reason, we're going to always set those variables and
    defer to the ansible layer the evaluation of the DashboardEnabled
    condition.

    [1] https://github.com/ansible/ansible/pull/73684

    Closes-Bug: #1949033
    Change-Id: I4ab2d20df22bdc9b140019f90d1370ef17f09294
    (cherry picked from commit b3863568ad75a3845389677b08a0d756e3c6e0b5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 16.0.0 release.