Error in keystone resources in tripleo-ansible with ansible 2.9

[Alfredo Moralejo]

Following error is happening in undercloud installation when testing TripleO with ansible 2.9:

https://logserver.rdoproject.org/14/25914/1/check/rdoinfo-tripleo-master-testing-centos-8-containers-multinode/34f7fb1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

TASK [tripleo_keystone_resources : Async assignment of Keystone user to roles] ***
Monday 16 March 2020 10:19:52 +0000 (0:00:00.141) 0:17:24.093 **********
changed: [undercloud] => (item={'heat': {'project': 'service'}})
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'str' object has no attribute 'get'
fatal: [undercloud]: FAILED! =>
  msg: Unexpected failure during module execution.
  stdout: ''

Is seems to not happen in all scenarios/featuresets, as jobs are passing in others.

A repository is available to test tripleo with the new version of ansible. To enable it, just need to add following .repo file:

https://logserver.rdoproject.org/14/25914/1/check/legacy-validate-buildsys-tags/22db5ad/buildset/repos/cloud8-openstack-ussuri-testing/temp-cloud8-openstack-ussuri-testing.repo

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/719443

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/719443
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=e86977ab48e4d7e66bda9060e51e396d4f6a85fe
Submitter: Zuul
Branch: master

commit e86977ab48e4d7e66bda9060e51e396d4f6a85fe
Author: yatinkarel <email address hidden>
Date: Mon Apr 13 17:38:14 2020 +0530

    Correctly set openstack_domains for ansible >= 2.9

    os_keystone_domain_info returns a dict which contains
    openstack_domains but currently that dict was considered
    as openstack_domains which is wrong, this patch fixes
    it.

    Closes-Bug: #1867608
    Change-Id: Ic063facf1f5f87c0e3a01803937a620a9b77d4b5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/720599

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720599
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=f89161592f48312b86ef3f94eea1bae52b6f6080
Submitter: Zuul
Branch: stable/train

commit f89161592f48312b86ef3f94eea1bae52b6f6080
Author: yatinkarel <email address hidden>
Date: Mon Apr 13 17:38:14 2020 +0530

    Correctly set openstack_domains for ansible >= 2.9

    os_keystone_domain_info returns a dict which contains
    openstack_domains but currently that dict was considered
    as openstack_domains which is wrong, this patch fixes
    it.

    Closes-Bug: #1867608
    Change-Id: Ic063facf1f5f87c0e3a01803937a620a9b77d4b5
    (cherry picked from commit e86977ab48e4d7e66bda9060e51e396d4f6a85fe)

[Jure Pečar (jpecar)]

After applying this fix to get rid of the initial bug, undercloud installation fails with:

TASK [tripleo-keystone-resources : Check Keystone user assignment to roles status] ***
Tuesday 21 April 2020 19:12:30 +0200 (0:00:04.655) 0:16:52.409 *********
changed: [undercloud] => (item={'ansible_loop_var': u'tripleo_keystone_resources_data_user', u'ansible_job_id': u'625621813083.484923', u'tripleo_keystone_resources_data_user': {u'ironic-inspector': {'project': u'service'}}, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/625621813083.484923'})
failed: [undercloud] (item={'ansible_loop_var': u'tripleo_keystone_resources_data_user', u'ansible_job_id': u'501724684534.484943', u'tripleo_keystone_resources_data_user': {u'heat_stack_domain_admin': {'domain': u'heat_stack'}}, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/root/.ansible_async/501724684534.484943'}) => {"ansible_job_id": "501724684534.484943", "ansible_loop_var": "tripleo_keystone_resources_user_role_async_result_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Must specify either a domain or project", "tripleo_keystone_resources_user_role_async_result_item": {"ansible_job_id": "501724684534.484943", "ansible_loop_var": "tripleo_keystone_resources_data_user", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/501724684534.484943", "started": 1, "tripleo_keystone_resources_data_user": {"heat_stack_domain_admin": {"domain": "heat_stack"}}}}

Content of the /root/.ansible_async/501724684534.484943 are:
{"msg": "Must specify either a domain or project", "failed": true, "invocation": {"module_args": {"auth_type": null, "domain": "3d544b22cce2462b95c5f4589e6947a5", "group": null, "region_name": null, "ca_cert": null, "availability_zone": null, "api_timeout": null, "auth": null, "project": null, "state": "present", "role": "admin", "user": "heat_stack_domain_admin", "timeout": 180, "interface": "public", "wait": true, "validate_certs": null, "client_cert": null, "client_key": null}}}

The way I see it is that with the above change, something later on doesn't get the domain passed as expected and fails.

[yatin (yatinkarel)]

<<< After applying this fix to get rid of the initial bug, undercloud installation fails with:
@jpechar which ansible version you are using? it should pass with ansible-2.9.7, specifically commit https://github.com/ansible/ansible/commit/a0491f804fc13f49b96ce7d2e5aaf798804c8827 is needed, can u please check.

[Jure Pečar (jpecar)]

Yes, that fixed it, thanks.
This is ansible 2.9.6 on CentOS 7.7.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 0.6.0 release.