Overcloud deployment fails with TLS-everywhere because of keytab permission issues

[Lance Bragstad]

TLS-e deployments require the undercloud to have a keytab it uses to authenticate to FreeIPA. The undercloud manages resources in FreeIPA for the overcloud infrastructure like hosts, services, and DNS entries.

A deployer either configures the undercloud keytab manually or passes a one-time password to the undercloud via it's configuration file, which enrolls the undercloud as a FreeIPA host, registers it, and allows it to fetch the keytab. Afterwards, the keytab permissions are updated so that it's readable by whatever user performs the overcloud installation.

With the removal of mistral from the undercloud, we've simplified some of the ansible variables to connect to the undercloud [0]. This change [1] causes some permission issues because tripleo-ipa assumed the tripleo-admin user and group as the ansible_ssh_user for the undercloud. Now that we're dealing with local connections [2]. This use can be anyone who is invoking the overcloud deploy.

You can recreate this by following the steps for TLS-e with tripleo-ipa, which are currently still under review [3].

The deployment fails during the overcloud installation when the undercloud attempts to enroll overcloud hosts to FreeIPA [4]. The failure manifests as a 401 because the user invoking the playbook (e.g., stack) doesn't have access to the keytab.

The expected result would be to successfully deploy the overcloud using the stack user. Note, this is only applicable on deployments not user deployed servers, since tripleo-admin is configured in the ansible inventory for those deployments.

[0] https://bugs.launchpad.net/tripleo/+bug/1884123
[1] https://review.opendev.org/#/c/736804/
[2] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/tls/undercloud-tls.yaml#L95-L99
[3] https://review.opendev.org/#/c/725607/
[4] https://opendev.org/x/tripleo-ipa/src/commit/c22fc8d07d7d6683895c9b141edcfd8350879a46/tripleo_ipa/roles/tripleo_ipa_registration/tasks/main.yml#L52-L57

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/740275

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/740275
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/9cb9618dc96130f716314625aa3fce4cf2a34ef4
Submitter: "Zuul (22348)"
Branch: master

commit 9cb9618dc96130f716314625aa3fce4cf2a34ef4
Author: Lance Bragstad <email address hidden>
Date: Thu Jul 9 08:05:40 2020 -0500

    Update undercloud TLS template with proper keytab group

    The undercloud supports enrolling itself as a FreeIPA client when
    configured to use TLS-everywhere. However, we recently hit a bug where
    the group permissions of the keytab were set to the old mistral user
    (tripleo-admin). This causes issues because more or what mistral did is
    being handled by ansible and the deployment user.

    This commit updates the group for the keytab to root, which the
    deployment users is already a member of. This keeps permission of the
    keytab strict but doesn't compromise usability by modifying the group in
    a way that requires the deployment user to re-authenticate to access it.

    Change-Id: Iacf5e6147f7ef02ba514b7dddc65383faa440826
    Closes-Bug: 1886870

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.1.0 release.