kolla-ansible fails to deploy heat if the heat_user_domain already exists in keystone

[Joseignacio-carretero]

I am deploying openstack Stein with kolla on Ubuntu 18.04. I have a keystone with data I want to keep from a previous Openstack installation. I already had a heat version installed and when I try to deploy Heat I get the following error:

+++ CURRENT_HEAT_DOMAIN_NAME='heat_user_domain
heat'
+++ [[ heat_user_domain != \\h\\e\\a\\t\\_\\u\\s\\e\\r\\_\\d\\o\\m\\a\\i\
\\
\\h\\e\\a\\t ]]
+++ openstack domain create heat_user_domain
Conflict occurred attempting to store domain - Duplicate Entry (HTTP 409) (Request-ID: req-c2e725bb-ebb8-40ef-9563-77977a6b959a)
", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /var/lib/kolla/config_files/heat.conf to /etc/heat/heat.conf", "INFO:__main__:Setting permission for /etc/heat/heat.conf", "INFO:__main__:Writing out command to execute", "++ cat /run_command", "+ CMD=heat-api", "+ ARGS=", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ [[ ! -d /var/log/kolla/heat ]]", "++ mkdir -p /var/log/kolla/heat", "+++ stat -c %a /var/log/kolla/heat", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/heat", "+++ whoami", "++ [[ heat == \\r\\o\\o\\t ]]", "++ . /usr/local/bin/kolla_heat_extend_start", "+++ [[ -n 0 ]]", "+++ heat-manage db_sync", "++++ openstack domain list", "++++ grep heat", "++++ awk '{print $4}'", "+++ CURRENT_HEAT_DOMAIN_NAME='heat_user_domain", "heat'", "+++ [[ heat_user_domain != \\h\\e\\a\\t\\_\\u\\s\\e\\r\\_\\d\\o\\m\\a\\i\
\\", "\\h\\e\\a\\t ]]", "+++ openstack domain create heat_user_domain", "Conflict occurred attempting to store domain - Duplicate Entry (HTTP 409) (Request-ID: req-c2e725bb-ebb8-40ef-9563-77977a6b959a)"], "stdout": "2019-08-12 09:46:37.324 17 INFO migrate.versioning.api [-] 70 -> 71... \u001b[00m

The work around is to delete the heat_user_domain in Keystone.

[Joseignacio-carretero (joseignacio-carretero)]

As a complementary information, this error also happen when Heat is deployed and "kolla-ansible -i inventory reconfigure" is run.

The work around is to remove the domain again:
openstack domain set --disable heat_user_domain
openstack domain delete heat_domain_admin

[Mark Goddard (mgoddard)]

This problem will keep happening because kolla uses the domain heat_user_domain domain. This is configured in heat.conf as stack_user_domain_name. If your old deployment used the domain 'heat' for heat, you need to delete that instead.

Clearly this could be more flexible in kolla/kolla-ansible, and allow the domain name to be configured.

[Mark Goddard (mgoddard)]

Just to clarify the bug description - the problem is not that heat_user_domain exists, it is that the heat domain exists.

Relevant code in kolla, docker/heat/heat-api/extend_start.sh:

#!/bin/bash

# Bootstrap and exit if KOLLA_BOOTSTRAP variable is set. This catches all cases
# of the KOLLA_BOOTSTRAP variable being set, including empty.
if [[ "${!KOLLA_BOOTSTRAP[@]}" ]]; then
    heat-manage db_sync
    CURRENT_HEAT_DOMAIN_NAME=$(openstack domain list | grep heat | awk '{print $4}')

    if [[ "heat_user_domain" != "$CURRENT_HEAT_DOMAIN_NAME" ]]; then
        openstack domain create heat_user_domain
        openstack user create --domain heat_user_domain heat_domain_admin --password ${HEAT_DOMAIN_ADMIN_PASSWORD}
        openstack role add --domain heat_user_domain --user-domain heat_user_domain --user heat_domain_admin admin
    fi
    exit 0
fi

Alternatively we could make the above match more strict, to match on heat_user_domain exactly.

[Joseignacio-carretero (joseignacio-carretero)]

Yes, when "heat_user_domain" exists and heat installation fails because of that. Both kolla-ansible deploy and kolla-ansible reconfigure are failing for that reason. I think it would be nice checking if the domain exists and if so, using the existent domain instead of trying to create it.

[Mark Goddard (mgoddard)]

The problem isn't that heat_user_domain exists. The problem is that heat_user_domain and another domain called heat both exist, which breaks our check to see if heat_user_domain exists.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/711923

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711923
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=5e946cac490077011709c7587dc5e0b06927464f
Submitter: Zuul
Branch: master

commit 5e946cac490077011709c7587dc5e0b06927464f
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

    Checks for heat_user_domain explicitly

    The script should check for heat_user_domain explicitly instead of matching on "heat"

    Closes-Bug: #1839824
    Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/758058

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/758059

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/758060

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758059
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=353d7cb279d91fe48a73ccdfcddf99f87f511316
Submitter: Zuul
Branch: stable/train

commit 353d7cb279d91fe48a73ccdfcddf99f87f511316
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

    Checks for heat_user_domain explicitly

    The script should check for heat_user_domain explicitly instead of matching on "heat"

    Closes-Bug: #1839824
    Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856
    (cherry picked from commit 5e946cac490077011709c7587dc5e0b06927464f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758060
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=bb62d2559fda8431c6e9721ae6ca6b6d0755921c
Submitter: Zuul
Branch: stable/stein

commit bb62d2559fda8431c6e9721ae6ca6b6d0755921c
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

    Checks for heat_user_domain explicitly

    The script should check for heat_user_domain explicitly instead of matching on "heat"

    Closes-Bug: #1839824
    Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856
    (cherry picked from commit 5e946cac490077011709c7587dc5e0b06927464f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758058
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=027016e98f72102718b2a7b5215be6ea4f03725f
Submitter: Zuul
Branch: stable/ussuri

commit 027016e98f72102718b2a7b5215be6ea4f03725f
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

    Checks for heat_user_domain explicitly

    The script should check for heat_user_domain explicitly instead of matching on "heat"

    Closes-Bug: #1839824
    Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856
    (cherry picked from commit 5e946cac490077011709c7587dc5e0b06927464f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 8.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 9.3.0 release.