openstack-tox-tht ci failing with sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error

[Michele Baldessari]

Seen on https://review.opendev.org/#/c/741610/. The complete failure from https://zuul.opendev.org/t/openstack/build/66c0569803df4b4c86c3e6f5889b88a3/log/job-output.txt is:

2020-08-12 08:33:07.466981 | ubuntu-bionic |

2020-08-12 08:33:07.467000 | ubuntu-bionic | TASK [tripleo_upgrade_hiera : ensure tripleo-upgrade hiera file exists] ********

2020-08-12 08:33:07.467019 | ubuntu-bionic | included: /home/zuul/src/opendev.org/openstack/tripleo-heat-templates/tripleo_heat_templates/tests/tripleo-ansible/tripleo-ansible/tripleo_ansible/roles/tripleo_upgrade_hiera/tasks/create-tripleo-upgrade-file.yml for localhost

2020-08-12 08:33:07.467043 | ubuntu-bionic |

2020-08-12 08:33:07.467063 | ubuntu-bionic | TASK [tripleo_upgrade_hiera : create the directory for hiera file] *************

2020-08-12 08:33:07.467082 | ubuntu-bionic | fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

2020-08-12 08:33:07.467101 | ubuntu-bionic | localhost : ok=19  changed=6  unreachable=0 failed=1  skipped=70  rescued=0 ignored=0

2020-08-12 08:33:07.467120 | ubuntu-bionic |

2020-08-12 08:33:07.467139 | ubuntu-bionic |

2020-08-12 08:33:07.467158 | ubuntu-bionic | PLAY RECAP *********************************************************************

2020-08-12 08:33:07.467177 | ubuntu-bionic | localhost : ok=19  changed=6  unreachable=0 failed=1  skipped=70  rescued=0 ignored=0

2020-08-12 08:33:07.467196 | ubuntu-bionic |

2020-08-12 08:33:07.467215 | ubuntu-bionic | failed: 2

2020-08-12 08:33:07.467240 | ubuntu-bionic | - generated html file: file:///home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/tht/log/reports.html -

2020-08-12 08:33:07.467262 | ubuntu-bionic | =========================== short test summary info ============================

2020-08-12 08:33:07.467281 | ubuntu-bionic | FAILED tripleo_heat_templates/tests/test_tht_ansible_syntax.py::test_tht_ansible_syntax

2020-08-12 08:33:07.467300 | ubuntu-bionic | ======================== 1 failed in 158.84s (0:02:38) =========================

[Marios Andreou (marios-b)]

14:52 < marios|ruck> bandini: np but i haven't seen this one yet thanks for filing it
14:53 < marios|ruck> bandini: might not be a general thing not sure yet
                     https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-tht&project=openstack/tripleo-heat-templates

[Michele Baldessari (michele)]

So the error gets triggered in tripleo_ansible/roles/tripleo_upgrade_hiera/tasks/create-tripleo-upgrade-file.yml:
- name: create the directory for hiera file
  file:
    path: "{{ tripleo_upgrade_hiera_file | dirname }}"
    owner: "root"
    group: "root"
    mode: 0755
    state: directory
  become: true

It seems that the way we invoke the test, sudo is not configured and so this fails. My initial thinking is that we added this test lately and the other upgrade_tasks in tht are not triggering it?

[Lance Bragstad (lbragstad)]

I'm seeing this on stable/train environments using RDO packages during scale down tasks [0]. The tripleo-ipa project uses become: true to cleanup hosts in FreeIPA when a node is deleted. The keytab we use to authenticate to FreeIPA requires root [1].

(undercloud) [stack@undercloud ~]$ openstack server list
+--------------------------------------+-------------------------+--------+------------------------+----------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------------+--------+------------------------+----------------+-----------+
| 4ea9059b-f91e-4b16-a4af-70d96acd372e | overcloud-controller-0 | ACTIVE | ctlplane=192.168.24.18 | overcloud-full | baremetal |
| 624942fd-9855-4bca-87e5-b90d63917fee | overcloud-novacompute-0 | ACTIVE | ctlplane=192.168.24.22 | overcloud-full | baremetal |
| 0c12414c-6c3e-4cbd-89a1-30eb19ac1aed | overcloud-novacompute-1 | ACTIVE | ctlplane=192.168.24.6 | overcloud-full | baremetal |
+--------------------------------------+-------------------------+--------+------------------------+----------------+-----------+
(undercloud) [stack@undercloud ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 6de764db-8b5f-45c2-910f-e1ed4cdbeedc | overcloud | c144927514e949b3936762c25f3dff28 | CREATE_COMPLETE | 2020-08-11T22:02:53Z | None |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
(undercloud) [stack@undercloud ~]$ openstack overcloud node delete --stack overcloud --yes 0c12414c-6c3e-4cbd-89a1-30eb19ac1aed
Deleting the following nodes from stack overcloud:
- 0c12414c-6c3e-4cbd-89a1-30eb19ac1aed
Waiting for messages on queue 'tripleo' with no timeout.
Config downloaded at /var/lib/mistral/overcloud
Inventory generated at /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml
Running ansible playbook at /var/lib/mistral/overcloud/scale_playbook.yaml. See log file at /var/lib/mistral/overcloud/ansible.log for progress. ...

PLAY [Clear cached facts] ******************************************************

TASK [Gathering Facts] *********************************************************
Wednesday 12 August 2020 12:59:20 +0000 (0:00:00.168) 0:00:00.168 ******
[WARNING]: Failure using method (v2_runner_on_start) in callback plugin
(<ansible.plugins.callback.tripleo.CallbackModule object at 0x7f6b935523d0>):
'show_per_host_start'
ok: [overcloud-novacompute-1]

PLAY [Gather facts from undercloud] ********************************************
skipping: no hosts matched

PLAY [Gather facts from overcloud] *********************************************

TASK [...

[Michele Baldessari (michele)]

So the issue was seen in my CI review for the following reasons:
- this tht tox renders a bunch of yaml files
- then somehow simulates the ansible bit by just importing them but doing nothing (it does so by setting step to 9999)
- now the problem is that in my post_upgrade task i run some commands without putting them under the step==something condition and so this ansible test will actually run things and fail

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/745952

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/745954

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/745955

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/745956

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lance Bragstad (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/745952
Reason: Abandoning in favor of a proper backport https://review.opendev.org/#/c/745956/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745954
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1547fc8e30df3745c615d10653e9febbbb0d37bc
Submitter: Zuul
Branch: master

commit 1547fc8e30df3745c615d10653e9febbbb0d37bc
Author: Lance Bragstad <email address hidden>
Date: Wed Aug 12 14:30:16 2020 -0500

    Fix delegation with FreeIPA cleanup

    Previously, we were delegating the IPA cleanup role to the undercloud
    via localhost. This is because the keytab used to authenticate to
    FreeIPA and perform the cleanup of host entries during scale down is on
    the undercloud. However, when using train, ansible is invoked from the
    mistral container when using `delegate_to: localhost`. In this case,
    you'll end up with a privilege escalation error:

      "sudo: unable to open /run/sudo/ts/mistral: Permission denied\nsudo: a password is required\n",

    This is because the mistral container doesn't have passwordless sudo,
    resulting in a failed privilege escalation.

    Instead, we should make sure we delegate this task to the Undercloud,
    where we know the tripleo-admin user is setup properly.

    Change-Id: I844f78c520d7b507d906faf7242e72dd717f9cb5
    Related-Bug: 1891317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745955
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b245565d7197437cd9186e97eb7d21aeb33bb349
Submitter: Zuul
Branch: stable/ussuri

commit b245565d7197437cd9186e97eb7d21aeb33bb349
Author: Lance Bragstad <email address hidden>
Date: Wed Aug 12 14:30:16 2020 -0500

    Fix delegation with FreeIPA cleanup

    Previously, we were delegating the IPA cleanup role to the undercloud
    via localhost. This is because the keytab used to authenticate to
    FreeIPA and perform the cleanup of host entries during scale down is on
    the undercloud. However, when using train, ansible is invoked from the
    mistral container when using `delegate_to: localhost`. In this case,
    you'll end up with a privilege escalation error:

      "sudo: unable to open /run/sudo/ts/mistral: Permission denied\nsudo: a password is required\n",

    This is because the mistral container doesn't have passwordless sudo,
    resulting in a failed privilege escalation.

    Instead, we should make sure we delegate this task to the Undercloud,
    where we know the tripleo-admin user is setup properly.

    Change-Id: I844f78c520d7b507d906faf7242e72dd717f9cb5
    Related-Bug: 1891317
    (cherry picked from commit 1547fc8e30df3745c615d10653e9febbbb0d37bc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745956
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d0c5bcac80f30364aafbf63b3f75a302ffcca216
Submitter: Zuul
Branch: stable/train

commit d0c5bcac80f30364aafbf63b3f75a302ffcca216
Author: Lance Bragstad <email address hidden>
Date: Wed Aug 12 14:30:16 2020 -0500

    Fix delegation with FreeIPA cleanup

    Previously, we were delegating the IPA cleanup role to the undercloud
    via localhost. This is because the keytab used to authenticate to
    FreeIPA and perform the cleanup of host entries during scale down is on
    the undercloud. However, when using train, ansible is invoked from the
    mistral container when using `delegate_to: localhost`. In this case,
    you'll end up with a privilege escalation error:

      "sudo: unable to open /run/sudo/ts/mistral: Permission denied\nsudo: a password is required\n",

    This is because the mistral container doesn't have passwordless sudo,
    resulting in a failed privilege escalation.

    Instead, we should make sure we delegate this task to the Undercloud,
    where we know the tripleo-admin user is setup properly.

    Change-Id: I844f78c520d7b507d906faf7242e72dd717f9cb5
    Related-Bug: 1891317
    (cherry picked from commit 1547fc8e30df3745c615d10653e9febbbb0d37bc)

[Marios Andreou (marios-b)]

looks like the fix this is merged back to train [1] and can't see any more examples [2]

moving this fix-released please move back if you disagree and there is more work to be tracked here.

[1] https://review.opendev.org/745956
[2] https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-tht#