Fix in 1649381 breaks rendering of block headers

[Alexey 'Fenuks' Rusetsky]

Removal of second re-templating of template in https://opendev.org/openstack/ansible-config_template/commit/e528ed0e9e9f3d3fcb2f33ddc5d175faf72094ac#diff-7841482f458540098fc6accfb90a21c1996b343e prevents rendering of header as recommended in https://docs.openstack.org/openstack-ansible/2023.1/user/ceph/swift.html.

Without second re-template the INI block header is inserted into ceph.conf as is:

[client.rgw.{{ hostvars[inventory_hostname]['ansible_facts']['hostname'] }}.rgw0]

instead of

[client.rgw.st0-ceph-mon-container-0e8b13a1.rgw0]

[Dmitriy Rabotyagov (noonedeadpunk)]

I can confirm that and do see output you're talking about in our CI results as well. Though it appears, we do not test RGW deployment there, only RBD part is tested.

[Alexey 'Fenuks' Rusetsky (fenuks-uh)]

I can confirm that pasting back that part from line 956 to 959 of plugins/action/config_template.py rendered all the configs correctly for me.
Perhaps that is the reason it was done in the first place.

[Dmitriy Rabotyagov (noonedeadpunk)]

Well, it's kinda weird, since all other variables from both template and override are rendered properly, except when it's a section name. So that seems to be some corner case, which hopefully has more elegant solution...

So override like that:

ceph_conf_overrides_rgw:
  "client.rgw.{{ hostvars[inventory_hostname]['ansible_facts']['hostname'] }}.rgw0":
    # OpenStack integration with Keystone
    rgw_keystone_url: "{{ keystone_service_adminuri }}"
    rgw_keystone_api_version: 3
    rgw_keystone_admin_user: "{{ radosgw_admin_user }}"
    rgw_keystone_admin_password: "{{ radosgw_admin_password }}"
    rgw_keystone_admin_project: "{{ radosgw_admin_tenant }}"
    rgw_keystone_admin_domain: default
    rgw_keystone_accepted_roles: 'member, admin, swiftoperator'
    rgw_keystone_implicit_tenants: 'true'
    rgw_swift_account_in_url: 'true'
    rgw_swift_versioning_enabled: 'true'
    rgw_enable_apis: 'swift, s3'
    rgw_s3_auth_use_keystone: 'true'

Get renderred as:

[client.rgw.{{ hostvars[inventory_hostname]['ansible_facts']['hostname'] }}.rgw0]
rgw_keystone_url = http://172.29.236.101:5000
rgw_keystone_api_version = 3
rgw_keystone_admin_user = radosgw
rgw_keystone_admin_password = 0d5136520095f78cc8f5773af79f305df35644617b07dff7a918796
rgw_keystone_admin_project = service
rgw_keystone_admin_domain = default
rgw_keystone_accepted_roles = member, admin, swiftoperator
rgw_keystone_implicit_tenants = true
rgw_swift_account_in_url = true
rgw_swift_versioning_enabled = true
rgw_enable_apis = swift, s3
rgw_s3_auth_use_keystone = true

[Dmitriy Rabotyagov (noonedeadpunk)]

Ok, so it appears to be a known and intentional limitation on how ansible intends to render/process mappings: https://github.com/ansible/ansible/issues/17324#issuecomment-685102595

[Alexey 'Fenuks' Rusetsky (fenuks-uh)]

Looks like it is also the reason for https://bugs.launchpad.net/openstack-ansible/+bug/1812245

Headers are inserted as-is and do not match existing or already templated parts, then second run templates them, but does not merge them with existing ones.

[Dmitriy Rabotyagov (noonedeadpunk)]

Yep, that sounds correct.

The more I look into the issue, the more I find current config_template module behaviour correct, or well, expected according to basic Ansible logic.

By far the best solution would be to use a workaround when defining mappings with keys as variables, like to what was proposed here:
https://github.com/mrlesmithjr/ansible-netplan/issues/17#issuecomment-712661518

I've tested common approach and looks like it's working properly.

So I'm inclined to patch documentation and defaults in group_vars to reflect that, while leaving current behavior of the module. As with #1812245 in mind - it's wasn't giving expected result in fact anyway.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/904741

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ansible-config_template/+/904744

[Alexey 'Fenuks' Rusetsky (fenuks-uh)]

https://review.opendev.org/c/openstack/openstack-ansible/+/904741 — this approach gives me the expected config (without changes to config_tempalte). Although rewriting OSA configs this way is very error-prone.

It also merges blocks correctly now, so this approach also avoids #1812245.

I suppose it could use a mention in releasenotes, as it's not the most obvious transition.

[Dmitriy Rabotyagov (noonedeadpunk)]

Yeah, I don't fully like defining bug structures this way, but other then that would be patching Ansible I assume, which they reject as it's "as designed". I don't see any clear way of patching config_template without bringing any other regression, since content appears in the module already rendered, as I assume it's resolved somewhere earlier.

Yeah, good point about release note, will take care of it.

[Alexey 'Fenuks' Rusetsky (fenuks-uh)]

Solution is working and is simple enough to implement by deployers without waiting from upstream, I'd say it's good enough.

If relevant, it all happened to me on upgrade from 27.2.0 to 27.3.0 (stable/2023.1).

[Dmitriy Rabotyagov (noonedeadpunk)]

Other potential way around would be to use older version of config_template collection, by creating /etc/openstack_deploy/user-collection-requirements.yml with the following content, and re-run ./scripts/bootstrap-ansible.sh:

collections:

- name: openstack.config_template
  source: https://opendev.org/openstack/ansible-config_template
  type: git
  version: 2.0.0

[Alexey 'Fenuks' Rusetsky (fenuks-uh)]

Can I recommend mentioning ceph-rgw config explicitly in releasenote?

It is the only supplied and documented override with templated dictionary key, so chances are hight it is the one that breaks for most people.
And releasenotes are what people check (at least I do) for upgrade points to implement before starting the upgrade process.

[Dmitriy Rabotyagov (noonedeadpunk)]

I've added another `fixes` note. Do you think that should be enough, or still better to add RGW to the `issues` section?

[Alexey 'Fenuks' Rusetsky (fenuks-uh)]

Yes, that "fixes" block would've been enough for me.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ansible-config_template/+/904744
Committed: https://opendev.org/openstack/ansible-config_template/commit/ca25191181ef1d1269afe95ad0911bac8d4d0610
Submitter: "Zuul (22348)"
Branch: master

commit ca25191181ef1d1269afe95ad0911bac8d4d0610
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Jan 4 13:42:30 2024 +0100

    [doc] Document limitation and workaround for variables in mapping keys

    Related-Bug: #2048036
    Change-Id: Iadaab9c1d9af63bb32a2f9f6977b08fa7f643165

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/904741
Committed: https://opendev.org/openstack/openstack-ansible/commit/4203aa26c6a0ae71ad9113f2a8dd54efbdd72f63
Submitter: "Zuul (22348)"
Branch: master

commit 4203aa26c6a0ae71ad9113f2a8dd54efbdd72f63
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Jan 4 13:09:39 2024 +0100

    Modify RGW client format

    With changes to config_template module that restored usage of {% raw %} tags [1]
    renderring of mapping keys, if they're defined as variables, was broken.

    Ansible, by design [2], does not render mapping keys. Moreover, it was not
    working as intended anyway, since renderring happened in post-copy stage
    so same records were not merged together, which resulted in #1812245

    As such behaviour is expected by Ansible design, instead of adding some
    workaround in config_template module, I suggest working around issue
    by defining troublesome mapping with Jinja, that will allow it to render properly.

    [1] https://review.opendev.org/c/openstack/ansible-config_template/+/881887
    [2] https://github.com/ansible/ansible/issues/17324#issuecomment-685102595

    Closes-Bug: #2048036
    Related-Bug: #1812245
    Change-Id: I8a32736239c6326d817c620451799c13d5d8938c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/904855

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/904856

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/904855
Committed: https://opendev.org/openstack/openstack-ansible/commit/1d878e6c538da421a712edcda91da548ed865859
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 1d878e6c538da421a712edcda91da548ed865859
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Jan 4 13:09:39 2024 +0100

    Modify RGW client format

    With changes to config_template module that restored usage of {% raw %} tags [1]
    renderring of mapping keys, if they're defined as variables, was broken.

    Ansible, by design [2], does not render mapping keys. Moreover, it was not
    working as intended anyway, since renderring happened in post-copy stage
    so same records were not merged together, which resulted in #1812245

    As such behaviour is expected by Ansible design, instead of adding some
    workaround in config_template module, I suggest working around issue
    by defining troublesome mapping with Jinja, that will allow it to render properly.

    [1] https://review.opendev.org/c/openstack/ansible-config_template/+/881887
    [2] https://github.com/ansible/ansible/issues/17324#issuecomment-685102595

    Closes-Bug: #2048036
    Related-Bug: #1812245
    Change-Id: I8a32736239c6326d817c620451799c13d5d8938c
    (cherry picked from commit 4203aa26c6a0ae71ad9113f2a8dd54efbdd72f63)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/904856
Committed: https://opendev.org/openstack/openstack-ansible/commit/dbc265ce59d04a5a5321f5a4c27c46bf336692c4
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit dbc265ce59d04a5a5321f5a4c27c46bf336692c4
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Jan 4 13:09:39 2024 +0100

    Modify RGW client format

    With changes to config_template module that restored usage of {% raw %} tags [1]
    renderring of mapping keys, if they're defined as variables, was broken.

    Ansible, by design [2], does not render mapping keys. Moreover, it was not
    working as intended anyway, since renderring happened in post-copy stage
    so same records were not merged together, which resulted in #1812245

    As such behaviour is expected by Ansible design, instead of adding some
    workaround in config_template module, I suggest working around issue
    by defining troublesome mapping with Jinja, that will allow it to render properly.

    [1] https://review.opendev.org/c/openstack/ansible-config_template/+/881887
    [2] https://github.com/ansible/ansible/issues/17324#issuecomment-685102595

    Closes-Bug: #2048036
    Related-Bug: #1812245
    Change-Id: I8a32736239c6326d817c620451799c13d5d8938c
    (cherry picked from commit 4203aa26c6a0ae71ad9113f2a8dd54efbdd72f63)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 28.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 27.4.0 release.