Failed to connect to the host via ssh: Warning: Permanently added '10.0.1.234' (ED25519) to the list of known hosts.\r\ncentos@10.0.1.234: Permission denied (publickey,gssapi-keyex,gssapi-with-mic - Deploy the FreeIPA server

[Ananya Banerjee]

periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master and periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master are failing with:

2022-01-05 08:57:34.025118 | primary | PLAY [Deploy the FreeIPA server] ***********************************************
2022-01-05 08:57:34.035492 | primary |
2022-01-05 08:57:34.035526 | primary | TASK [Gathering Facts] *********************************************************
2022-01-05 08:57:34.035696 | primary | Wednesday 05 January 2022 08:57:34 -0500 (0:00:00.139) 0:00:08.318 *****
2022-01-05 08:57:38.529443 | primary | fatal: [supplemental]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '10.0.1.234' (ED25519) to the list of known hosts.\r\ncentos@10.0.1.234: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).", "unreachable": true}

https://logserver.rdoproject.org/74/35174/14/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/a6684bc/job-output.txt

https://logserver.rdoproject.org/39/37739/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/c665158/job-output.txt

[Ananya Banerjee (frenzyfriday)]

Still hitting this

https://logserver.rdoproject.org/39/37739/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/e17d01a/job-output.txt

https://logserver.rdoproject.org/74/35174/18/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/2222ab8/job-output.txt

[Ade Lee (alee-3)]

The issue appears to be that the magic user in the image has been changed from 'centos' to 'cloud_user'. I tried logging in as that user and I see the correct key in authorized_keys. There is no 'centos' user.

[Alan Pevec (apevec)]

Right, CS9 as RHEL upstream is now using the same kickstarts to build images, and in RHEL cloud images user was always cloud-user, this diff must have been already handled in CI to support RHEL8 vs CentOS 8 before?

[Ronelle Landy (rlandy)]

We never ran this test on RHEL before so this is our first hit on the issue

[Ronelle Landy (rlandy)]

Attempted change:

https://review.opendev.org/c/openstack/tripleo-quickstart/+/824644 DNM: Testing change supp user per ansible_distribution_major_version

[Ronelle Landy (rlandy)]

With ^^ patch, the job does start deploying the IPA server:

 https://logserver.rdoproject.org/54/36254/59/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/d16b733/logs/supplemental/home/cloud-user/deploy_freeipa.log.txt.gz

Complete!
+ dnf install -yq ipa-server ipa-server-dns curl epel-release iptables
Error: Unable to find a match: epel-release

^^ latest error

[Alan Pevec (apevec)]

uhm why epel-release, where is that coming from? We are avoiding deps from EPEL, if something from EPEL is required, we need to add it to RDO deps.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824705

[Ananya Banerjee (frenzyfriday)]

Went further, now failing at:

TASK [freeipa-setup : Run the tripleo-ipa preparation script] ******************
2022-01-16 17:04:56.449175 | primary | Sunday 16 January 2022 17:04:56 -0500 (0:00:03.092) 0:11:10.955 ********
2022-01-16 17:05:23.837629 | primary | fatal: [supplemental]: FAILED! => {"changed": true, "cmd": "set -o pipefail && ~cloud-user/ipa_prep.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > ~cloud-user/ipa_prep.sh.log\n", "delta": "0:00:25.895837", "end": "2022-01-16 17:05:23.619910", "msg": "non-zero return code", "rc": 4, "start": "2022-01-16 17:04:57.724073", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

[Ananya Banerjee (frenzyfriday)]

2022-01-16 17:05:23 | ERROR! couldn't resolve module/action 'ipa_role'. This often indicates a misspelling, missing collection, or incorrect module path.
2022-01-16 17:05:23 |
2022-01-16 17:05:23 | The error appears to be in '/usr/share/ansible/tripleo-playbooks/ipa-server-create-role.yaml': line 104, column 7, but may
2022-01-16 17:05:23 | be elsewhere in the file depending on the exact syntax problem.
2022-01-16 17:05:23 |
2022-01-16 17:05:23 | The offending line appears to be:
2022-01-16 17:05:23 |
2022-01-16 17:05:23 |
2022-01-16 17:05:23 | - name: add nova host manager role
2022-01-16 17:05:23 | ^ here

https://logserver.rdoproject.org/54/36254/62/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/df3ab85/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824705
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/3fc8e89a2ee92f271ce1470cfa4e9a4122f88ee4
Submitter: "Zuul (22348)"
Branch: master

commit 3fc8e89a2ee92f271ce1470cfa4e9a4122f88ee4
Author: frenzyfriday <email address hidden>
Date: Fri Jan 14 18:17:51 2022 +0530

    Update freeipa setup scripts for CentOS Stream 9

    - epel release is used here to install haveged. But haveged can be installed
      as rdo dependency.
    - install ansible-core instead of ansible for >= CS9
    - install community.general when using ansible-core, to have 'ipa_role'
      module available.

    Related-Bug: 1956563
    Change-Id: I62c16e60f164b564dd8474c049205e7141941f7f
    Signed-off-by: Douglas Viroel <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/824644
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/cf48a79c8fa5f834e0ebac93d1d336c15fccf1b2
Submitter: "Zuul (22348)"
Branch: master

commit cf48a79c8fa5f834e0ebac93d1d336c15fccf1b2
Author: Ronelle Landy <email address hidden>
Date: Thu Jan 13 15:03:37 2022 -0500

    Change supp user per ansible_distribution_major_version

    When running on CentOS Stream 9 distribution, the supplemental
    user should be 'cloud-user', instead of 'centos'

    Closes-Bug: #1956563
    Change-Id: Id90c0bcc2f15a742b83b834acc75231ca4d44a83