kinit: Client '<email address hidden>' not found in Kerberos database while getting initial credentials

[Ananya Banerjee]

tripleo-ci-centos-9-standalone-on-multinode-ipa is failing in TASK [tripleo.operator.tripleo_deploy : Standalone deploy] with

TASK | get a new keytab
2022-03-03 21:04:38.981310 | fa163e45-c401-66e5-bc85-00000000122e | FATAL | get a new keytab | undercloud | error={"changed": true, "cmd": ["kinit", "-kt", "/etc/novajoin/krb5.keytab", "nova/standalone-0.ooo.test"], "delta": "0:00:00.010778", "end": "2022-03-03 21:04:38.953994", "msg": "non-zero return code", "rc": 1, "start": "2022-03-03 21:04:38.943216", "stderr": "kinit: Client '<email address hidden>' not found in Kerberos database while getting initial credentials", "stderr_lines": ["kinit: Client '<email address hidden>' not found in Kerberos database while getting initial credentials"], "stdout": "", "stdout_lines": []}
2022-03-03 21:04:38.982511 | fa163e45-c401-66e5-bc85-00000000122e | TIMING | tripleo_ipa_registration : get a new keytab | undercloud | 0:03:37.784111 | 0.26s

logs:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9d5/830987/10/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/9d5a3e0/logs/undercloud/home/zuul/standalone_deploy.log
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_faa/828144/4/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/faa42d1/logs/undercloud/home/zuul/standalone_deploy.log

[Ananya Banerjee (frenzyfriday)]

There is also another kerberos issue going on: https://bugs.launchpad.net/tripleo/+bug/1963545 but might not be the same issue

[Ananya Banerjee (frenzyfriday)]

We see this stating from 2022-03-03 13:15:59

[Ananya Banerjee (frenzyfriday)]

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_faa/828144/4/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/faa42d1/logs/undercloud/home/zuul/undercloud-ipa-install.log

[Douglas Viroel (dviroel)]

Failing here:

TASK [tripleo_ipa_setup : add Nova Host Manager role] **************************
task path: /usr/share/ansible/roles/tripleo_ipa_setup/tasks/setup.yml:80
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: zuul
<127.0.0.1> EXEC /bin/sh -c 'echo ~zuul && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/zuul/.ansible/tmp `"&& mkdir "` echo /home/zuul/.ansible/tmp/ansible-tmp-1646334669.7392476-18929-55959497267787 `" && echo ansible-tmp-1646334669.7392476-18929-55959497267787="` echo /home/zuul/.ansible/tmp/ansible-tmp-1646334669.7392476-18929-55959497267787 `" ) && sleep 0'
redirecting (type: modules) ansible.builtin.ipa_role to community.general.ipa_role
Using module file /usr/share/ansible/collections/ansible_collections/community/general/plugins/modules/ipa_role.py
<127.0.0.1> PUT /home/zuul/.ansible/tmp/ansible-local-16125gu58pmrz/tmps3f_oz8n TO /home/zuul/.ansible/tmp/ansible-tmp-1646334669.7392476-18929-55959497267787/AnsiballZ_ipa_role.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/zuul/.ansible/tmp/ansible-tmp-1646334669.7392476-18929-55959497267787/ /home/zuul/.ansible/tmp/ansible-tmp-1646334669.7392476-18929-55959497267787/AnsiballZ_ipa_role.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-kdyicwvkfxborvjsdltsrmujsyyktrux ; IPA_USER=admin IPA_HOST=ipa.ooo.test IPA_PASS=fce95318204114530f31f885c9df588f /usr/bin/python3.9 /home/zuul/.ansible/tmp/ansible-tmp-1646334669.7392476-18929-55959497267787/AnsiballZ_ipa_role.py'"'"' && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/zuul/.ansible/tmp/ansible-tmp-1646334669.7392476-18929-55959497267787/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "cn": "Nova Host Manager",
            "description": "Nova Host Manager",
            "group": null,
            "host": null,
            "hostgroup": null,
            "ipa_host": "ipa.ooo.test",
            "ipa_pass": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "ipa_port": 443,
            "ipa_prot": "https",
            "ipa_timeout": 10,
            "ipa_user": "admin",
            "name": "Nova Host Manager",
            "privilege": [
                "Nova Host Management"
            ],
            "service": null,
            "state": "present",
            "user": null,
            "validate_certs": true
        }
    },
    "msg": "login: HTTP Error 500: Internal Server Error"
}

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_faa/828144/4/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/faa42d1/logs/undercloud/home/zuul/undercloud-ipa-install.log

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/831942

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832116

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/831942
Committed: https://opendev.org/openstack/tripleo-ci/commit/3de35c67026f664c2dc7fca0bce8500664a5a0b0
Submitter: "Zuul (22348)"
Branch: master

commit 3de35c67026f664c2dc7fca0bce8500664a5a0b0
Author: frenzyfriday <email address hidden>
Date: Fri Mar 4 19:23:50 2022 +0530

    Make tripleo-ci-centos-9-standalone-on-multinode-ipa non voting till 1963675 is fixed

    Related-Bug: 1963675
    Change-Id: I5ac2dedfad79a524c3a03435545633f0a4c16ae1

[Ananya Banerjee (frenzyfriday)]

We see this in periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master also.
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/f05c548/logs/undercloud/home/zuul/ansible.log.txt.gz

[Sandeep Yadav (sandeepyadav93)]

Temporary workarounds:-

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832116 and https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832309

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832116
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/8175a19fd9e4cccd36c5ede80906023da0f43ee6
Submitter: "Zuul (22348)"
Branch: master

commit 8175a19fd9e4cccd36c5ede80906023da0f43ee6
Author: Dariusz Smigiel <email address hidden>
Date: Fri Mar 4 09:32:33 2022 -0800

    Temporarily freeze openssl

    Due to issues with latest openssl version use working version
    of openssl for ipa-multinode job.

    Related-Bug: 1963675
    Change-Id: I99c7ec5d5271745ca8d0856a8141787b99880161

[Alan Pevec (apevec)]

Did you try this workaround by lmiccini ?

"update-crypto-policies --set DEFAULT:SHA1" + reboot

[Alan Pevec (apevec)]

https://bugzilla.redhat.com/show_bug.cgi?id=2057471 was closed in RHEL 9 ipa-4.9.8-7.el9_0 in May 2022