template error at Copying over glance haproxy config

[Satish Patel]

I am deploying kolla-ansible latest release 2024.1 on Ubuntu 22.04 and getting strange error at following task

When I am running kolla-ansible -i multinode deploy

TASK [include_role : glance] *****************************************************************************************************************************************************************
task path: /usr/local/share/kolla-ansible/ansible/site.yml:168
redirecting (type: modules) ansible.builtin.firewalld to ansible.posix.firewalld

TASK [haproxy-config : Copying over glance haproxy config] ***********************************************************************************************************************************
task path: /usr/local/share/kolla-ansible/ansible/roles/haproxy-config/tasks/main.yml:2

The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ansible/plugins/action/template.py", line 152, in run
    resultant = templar.do_template(template_data, preserve_trailing_newlines=True, escape_backslashes=False, overrides=overrides)
  File "/usr/local/lib/python3.10/dist-packages/ansible/template/__init__.py", line 1010, in do_template
    res = myenv.concat(rf)
  File "/usr/local/lib/python3.10/dist-packages/ansible/template/native_helpers.py", line 83, in ansible_concat
    return ''.join([to_text(v) for v in nodes])
  File "/usr/local/lib/python3.10/dist-packages/ansible/template/native_helpers.py", line 83, in <listcomp>
    return ''.join([to_text(v) for v in nodes])
  File "<template>", line 436, in root
  File "/usr/local/lib/python3.10/dist-packages/ansible/template/__init__.py", line 381, in call
    return super().call(obj, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
  File "/usr/local/lib/python3.10/dist-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 193, in macro
  File "/usr/local/lib/python3.10/dist-packages/ansible/template/__init__.py", line 295, in wrapper
    ret = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/jinja2/filters.py", line 1030, in do_format
    return soft_str(value) % (kwargs or args)
  File "/usr/local/lib/python3.10/dist-packages/markupsafe/__init__.py", line 165, in __mod__
    return self.__class__(super().__mod__(value))
  File "/usr/local/lib/python3.10/dist-packages/markupsafe/__init__.py", line 370, in __str__
    return str(self.escape(self.obj))
  File "/usr/local/lib/python3.10/dist-packages/markupsafe/__init__.py", line 235, in escape
    rv = escape(s)
  File "/usr/local/lib/python3.10/dist-packages/markupsafe/__init__.py", line 45, in escape
    return Markup(_escape_inner(str(s)))
SystemError: <built-in function _escape_inner> returned NULL without setting an exception
failed: [os2-bos-ctrl01] (item={'key': 'glance-api', 'value': {'container_name': 'glance_api', 'group': 'glance-api', 'host_in_groups': True, 'enabled': True, 'image': 'registry.example.com/kolla/glance-api:2024.1-ubuntu-jammy', 'environment': {'http_proxy': '', 'https_proxy': '', 'no_proxy': 'localhost,127.0.0.1,10.0.52.11,10.0.52.10'}, 'privileged': False, 'volumes': ['/etc/kolla/glance-api/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'glance:/var/lib/glance/', '', 'kolla_logs:/var/log/kolla/', '', ''], 'dimensions': {}, 'healthcheck': {'interval': '30', 'retries': '3', 'start_period': '5', 'test': ['CMD-SHELL', 'healthcheck_curl http://10.0.52.11:9292'], 'timeout': '30'}, 'haproxy': {'glance_api': {'enabled': True, 'mode': 'http', 'external': False, 'port': '9292', 'frontend_http_extra': ['timeout client 6h'], 'backend_http_extra': ['timeout server 6h'], 'custom_member_list': ['server os2-bos-ctrl01 10.0.52.11:9292 check inter 2000 rise 2 fall 5', '']}, 'glance_api_external': {'enabled': True, 'mode': 'http', 'external': True, 'external_fqdn': 'openstack-bos-2.example.com', 'port': '9292', 'frontend_http_extra': ['timeout client 6h'], 'backend_http_extra': ['timeout server 6h'], 'custom_member_list': ['server os2-bos-ctrl01 10.0.52.11:9292 check inter 2000 rise 2 fall 5', '']}}}}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": {
        "key": "glance-api",
        "value": {
            "container_name": "glance_api",
            "dimensions": {},
            "enabled": true,
            "environment": {
                "http_proxy": "",
                "https_proxy": "",
                "no_proxy": "localhost,127.0.0.1,10.0.52.11,10.0.52.10"
            },
            "group": "glance-api",
            "haproxy": {
                "glance_api": {
                    "backend_http_extra": [
                        "timeout server 6h"
                    ],
                    "custom_member_list": [
                        "server os2-bos-ctrl01 10.0.52.11:9292 check inter 2000 rise 2 fall 5",
                        ""
                    ],
                    "enabled": true,
                    "external": false,
                    "frontend_http_extra": [
                        "timeout client 6h"
                    ],
                    "mode": "http",
                    "port": "9292"
                },
                "glance_api_external": {
                    "backend_http_extra": [
                        "timeout server 6h"
                    ],
                    "custom_member_list": [
                        "server os2-bos-ctrl01 10.0.52.11:9292 check inter 2000 rise 2 fall 5",
                        ""
                    ],
                    "enabled": true,
                    "external": true,
                    "external_fqdn": "openstack-bos-2.example.com",
                    "frontend_http_extra": [
                        "timeout client 6h"
                    ],
                    "mode": "http",
                    "port": "9292"
                }
            },
            "healthcheck": {
                "interval": "30",
                "retries": "3",
                "start_period": "5",
                "test": [
                    "CMD-SHELL",
                    "healthcheck_curl http://10.0.52.11:9292"
                ],
                "timeout": "30"
            },
            "host_in_groups": true,
            "image": "registry.example.com/kolla/glance-api:2024.1-ubuntu-jammy",
            "privileged": false,
            "volumes": [
                "/etc/kolla/glance-api/:/var/lib/kolla/config_files/:ro",
                "/etc/localtime:/etc/localtime:ro",
                "/etc/timezone:/etc/timezone:ro",
                "glance:/var/lib/glance/",
                "",
                "kolla_logs:/var/log/kolla/",
                "",
                ""
            ]
        }
    },
    "msg": "SystemError: <built-in function _escape_inner> returned NULL without setting an exception"
}

[Satish Patel (satish-txt)]

After downgrade MarkupSafe

# pip list | grep -i MarkupSafe
MarkupSafe 3.0.1

Solved after downgrade to 2.1.5

# pip install MarkupSafe==2.1.5

[Michal Nasiadka (mnasiadka)]

Have you tried MarkupSafe 3.0.2? (Just trying to understand which versions are affected and if it is fixed in 3.0.2)

[Jan Horstmann (janhorstmann)]

This has potentially been fixed in MarkupSafe 3.0.2

https://github.com/pallets/markupsafe/releases/tag/3.0.2
https://github.com/pallets/markupsafe/issues/472

I have not verified that yet, but maybe you can test upgrading MarkupSafe to 3.0.2

[Jan Horstmann (janhorstmann)]

I have tested MarkupSafe==3.0.2 and can no longer observe the issue