config_template is not fully python3 compatible

[Jonathan Rosser]

stable/queens will fail at /etc/ansible/roles/plugins/action/config_template.py", line 219, triggered by an override like this (and likely many others):

pip_global_conf_overrides.global.index-url: <url>

Line 219 includes reference to class "unicode" which is not defined in python3 but does work in python2. All py3 strings are unicode by default.

The sledgehammer approach to make a deploy work is to disable detection of python3 in bootstrap-ansible.sh

[Stuart Grace (stuartgrace)]

-vvv output:

The full traceback is:
Traceback (most recent call last):
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 130, in run
    res = self._execute()
  File "/opt/ansible-runtime/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 528, in _execute
    result = self._handler.run(task_vars=variables)
  File "/etc/ansible/roles/plugins/action/config_template.py", line 613, in run
    ignore_none_type=_vars.get('ignore_none_type', True)
  File "/etc/ansible/roles/plugins/action/config_template.py", line 310, in return_config_overrides_ini
    config.readfp(config_object)
  File "/usr/lib/python3.5/configparser.py", line 760, in readfp
    self.read_file(fp, source=filename)
  File "/usr/lib/python3.5/configparser.py", line 715, in read_file
    self._read(f, source)
  File "/etc/ansible/roles/plugins/action/config_template.py", line 219, in _read
    elif isinstance(cursect[optname], (str, unicode)):
NameError: name 'unicode' is not defined

[Logan V (loganv)]

So really the only place we call config_template in a py3 context is during bootstrap hosts, and it exposes issues like we saw with yaml overrides last year:
https://github.com/openstack/openstack-ansible-plugins/commit/2a286fb0fb0b1fc8fab2126837600c1c70f4e8de
https://github.com/openstack/openstack-ansible-plugins/commit/2d1dd94264e9406c0d4109e5877a2eb0237e06c2

Seems we have similar breakages for py3 with INI templates that are exposed when config_overrides are provided. We don't see it in the gate because config_overrides dicts are always empty in the gate which means none of the config_template override features are exercised there.

It also appears the py3 non-voting test in the plugins repo does not fail when using ini w/ config_overrides, so that is something we'll need to address also. (Random recent merged patch test result to follow):
http://logs.openstack.org/45/557545/1/check/openstack-ansible-python3-ubuntu-xenial-nv/494991a/job-output.txt.gz

Since the integrated gate is not useful to test our plugins compatibility with py3, and since we rely on py3 compat to some extent in the integrated repo, we should aim to get this py3 test in the plugins repo working and voting asap.

[Andy McCrae (andrew-mccrae)]

The py3 test is passing now, which suggests we need a better test :)
I'll make the py3 test voting though, but it'd be cool to know how to recreate that specific bug in py3.

[Andy McCrae (andrew-mccrae)]

So the job is now voting - this doesn't seem resolved yet since the unicode bit is still accurate.

Do you have an example of the conf overrides you were using to generate this error? I'd love to add them in to the testing to make it fail so we can know the fix will work - from my initial review it seems it should fail on byte strings since the py3 unicode went away, it should probably look like str, bytes for py3 - but again, would be good to test that and ensure it's working.

[Andy McCrae (andrew-mccrae)]

Ok I recreated it, should have a patch up soon

[Jonathan Rosser (jrosser)]

For reference, this was originally triggered by the following in user_variables.yml

pip_global_conf_overrides:
  global:
    index-url: http://pip.os.mist.rd.bbc.co.uk/root/pypi/+simple
    trusted-host: "pip.os.mist.rd.bbc.co.uk {{ internal_lb_vip_address }}"

[Andy McCrae (andrew-mccrae)]

@jrosser - thanks!
The PR I've added should work for fixing that up in py3 - if you could test it out that would be great. I've added in a test that adds in a unicode "+" to ensure we don't run into that issue again, without the change to the config_template.py the py3 test now fails.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/563585
Committed: https://git.openstack.org/cgit/openstack/ansible-config_template/commit/?id=fc1c9311a8e25fce011815f7d4a2dc7d41c99815
Submitter: Zuul
Branch: master

commit fc1c9311a8e25fce011815f7d4a2dc7d41c99815
Author: Andy McCrae <email address hidden>
Date: Mon Apr 23 12:45:08 2018 +0100

    Fix py3 unicode issue

    unicode is nolonger a type in py3 - instead the str type is used for
    unicode, and the bytes type would cover the str type from py2.

    To avoid having to version the code, we should try to use unicode,
    except the NameError and continue as though it is py3 if that fails.

    Additionally, this patch adds a test that will fail if you revert the
    config_template.py back to it's original in py3.

    Closes-bug: 1763422
    Change-Id: Ifda972caada27ade2d80f77b3df70568406226ff