upgrade yoga2antelope on ubuntu20.04 : os-heat-install fatal error

[Peter Struys]

Hi,

When running playbook os-heat-install.yml in our test-environment we get a fatal eror with "TASK [openstack.osa.service_setup : Add service project]". Error is "fatal: [controller01_heat_api_container-452bc50e -> controller01_utility_container-d99d11d8(172.29.238.52)]: FAILED! => {"attempts": 5, "changed": false, "extra_data": {"data": null, "details": null, "response": "None"}, "msg": "No Domain found for Default"}. We saw some previous errors & solutions for this playbook but that did not help us. All the other playbooks run without errors.

Antelope version is 27.4.2.

ansible [core 2.13.8]
python version = 3.8.10

# /etc/ansible/ansible_collections
Collection Version
------------------------- -------
ansible.netcommon 5.0.0
ansible.posix 1.5.0
ansible.utils 2.9.0
community.crypto 2.11.1
community.general 6.5.0
community.mysql 3.6.0
community.rabbitmq 1.2.3
gluster.gluster 1.0.2
openstack.cloud 2.0.0
openstack.config_template 2.1.0
openstack.osa 1.0.0
openvswitch.openvswitch 2.1.0

The full traceback is:
  File "/tmp/ansible_openstack.cloud.project_payload_ya9gub1d/ansible_openstack.cloud.project_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py", line 415, in __call__
    results = self.run()
  File "/tmp/ansible_openstack.cloud.project_payload_ya9gub1d/ansible_openstack.cloud.project_payload.zip/ansible_collections/openstack/cloud/plugins/modules/project.py", line 130, in run
  File "/tmp/ansible_openstack.cloud.project_payload_ya9gub1d/ansible_openstack.cloud.project_payload.zip/ansible_collections/openstack/cloud/plugins/modules/project.py", line 222, in _find
  File "/openstack/venvs/utility-27.4.2/lib/python3.8/site-packages/openstack/identity/v3/_proxy.py", line 204, in find_domain
    return self._find(_domain.Domain, name_or_id,
  File "/openstack/venvs/utility-27.4.2/lib/python3.8/site-packages/openstack/proxy.py", line 503, in _find
    return resource_type.find(
  File "/openstack/venvs/utility-27.4.2/lib/python3.8/site-packages/openstack/resource.py", line 2262, in find
    raise exceptions.ResourceNotFound(
fatal: [jommeke_heat_api_container-452bc50e -> jommeke_utility_container-d99d11d8(172.29.238.52)]: FAILED! => {
    "attempts": 5,
    "changed": false,
    "extra_data": {
        "data": null,
        "details": null,
        "response": "None"
    },
    "invocation": {
        "module_args": {
            "api_timeout": null,
            "auth": null,
            "auth_type": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "description": null,
            "domain": "Default",
            "domain_id": "Default",
            "endpoint_type": "admin",
            "extra_specs": null,
            "interface": "admin",
            "is_enabled": null,
            "name": "admin",
            "region_name": null,
            "sdk_log_level": "INFO",
            "sdk_log_path": null,
            "state": "present",
            "timeout": 180,
            "validate_certs": true,
            "wait": true
        }
    },
    "msg": "No Domain found for Default"
}

Any ideas ?

thank you
regards
Peter

[Peter Struys (peterstruys)]

I see a link with bug #2048209 ...

[Dmitriy Rabotyagov (noonedeadpunk)]

Hi,

I don't think it's really related.

Week or so ago we have discovered weird behavior in ansible-collections-openstack for openstack.cloud.identity_domain module, as supplying `default` as domain name there will result in renaming a domain from `Default` to `default`, which I believe is the root cause of this issue.

We had a discussion with collection maintainers, though it did not have any outcome:
https://meetings.opendev.org/irclogs/%23openstack-ansible-sig/%23openstack-ansible-sig.2024-04-23.log.html

Just to confirm, can you try these:
1. openstack domain show default -c id -c name

If id is "default" and name is also "default" (instead of expected "Default):
2. openstack domain set --name Default default

and re-run os-heat-install.yml playbook.

[Peter Struys (peterstruys)]

Hi Dmitriy,

That indeed solved the problem. The values were "id: default, name: default". Changed it as to your suggestion to "id: default, name: Default" and the playbook now happily says : "Playbook execution success".

In my production environment it is "id: default, name: Default". Never paid attention to this and I have no idea where the difference between production and test comes from.

Again, as allways, kudos for your excellent work and support.
I hope enough people appreciate your efforts for openstack-ansible :-).

regards
Peter

[Peter Struys (peterstruys)]

Extra : Production environment still on unmaintained/yoga.

[Peter Struys (peterstruys)]

It seems to me that some playbooks in setup-openstack.yml set the domain name back to lowercase. So, I had to switch it back to "Default" in time before the heat playbook was running.

[Dmitriy Rabotyagov (noonedeadpunk)]

If you will be able to spot which exact role does that change - it would be extremely helpful for us to workaround module behaviour.

Also we were reported previously, that it was part of federation setup, but name `default` was referenced in user_variables.

[Peter Struys (peterstruys)]

No problem. I will run the playbooks from setup-openstack one-by-one and check also within playbooks. I will get back to you on this.

[Peter Struys (peterstruys)]

This playbook for sure changes the domain name from "Default" to "default". I will check further if other playbooks do too. I extract the (changed) domain name from the db directly because the OpenStack client gives me mixed results with "openstack domain show default -c id -c name"

echo 'use keystone;select id,name from project where is_domain=1;' | ./mysql --table

TASK [os_keystone : Ensure domain which remote IDP users are mapped onto exists] *********************************************************************************************************
task path: /etc/ansible/roles/os_keystone/tasks/keystone_federation_sp_idp_setup.yml:21

changed: [controller01_keystone_container-e4e05fb2 -> controller01_utility_container-d99d11d8(172.29.238.52)] => (item={'domain': 'default', 'project': 'default_project', 'role': '_member_'}) => {
    "ansible_loop_var": "item",
    "changed": true,
    "domain": {
        "description": "The default domain",
        "id": "default",
        "is_enabled": true,
        "links": {
            "self": "https://172.29.236.199:5000/v3/domains/default"
        },
        "name": "default"
    },
    "invocation": {
        "module_args": {
            "api_timeout": null,
            "auth": null,
            "auth_type": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "description": null,
            "interface": "admin",
            "is_enabled": null,
            "name": "default",
            "region_name": null,
            "sdk_log_level": "INFO",
            "sdk_log_path": null,
            "state": "present",
            "timeout": 180,
            "validate_certs": false,
            "verify": false,
            "wait": true
        }
    },
    "item": {
        "domain": "default",
        "project": "default_project",
        "role": "_member_"
    }

[Dmitriy Rabotyagov (noonedeadpunk)]

Aha, federation one. We had report that during federation setup it happens, but you added very important detail to it.
Would be great if you could check if there's anything else and we can patch that asap

[Dmitriy Rabotyagov (noonedeadpunk)]

So, I think that in fact you'd need to adjust your "keystone_sp - trusted_idp_list - federated_identities" to contain "domain: Default" instead of `domain: default" to workaround the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/919563