failing task "service-ks-register : keystone | Creating services"

[Mark Goddard]

CI deploy jobs are currently broken on master. We get errors like the following:

TASK [service-ks-register : keystone | Creating services] **********************
task path: /home/zuul/src/opendev.org/openstack/kolla-ansible/ansible/roles/service-ks-register/tasks/main.yml:3
FAILED - RETRYING: keystone | Creating services (5 retries left).Result was: {
    "action": "os_keystone_service",
    "attempts": 1,
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "module_args": {
                "auth": {
                    "auth_url": "http://192.0.2.10:35357",
                    "domain_name": "default",
                    "password": "1zwBct8GtM44jb97UuM2l5vhcs85UzbA3snC3et3",
                    "project_name": "admin",
                    "user_domain_name": "default",
                    "username": "admin"
                },
                "cacert": "",
                "description": "Openstack Identity Service",
                "interface": "admin",
                "name": "keystone",
                "region_name": "RegionOne",
                "service_type": "identity"
            },
            "module_extra_vars": null,
            "module_name": "os_keystone_service",
            "timeout": 180,
            "user": null
        }
    },
    "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible-tmp-1583403053.4-202030654516731/AnsiballZ_os_keystone_service.py\", line 114, in <module>\n _ansiballz_main()\n File \"/tmp/ansible-tmp-1583403053.4-202030654516731/AnsiballZ_os_keystone_service.py\", line 106, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/tmp/ansible-tmp-1583403053.4-202030654516731/AnsiballZ_os_keystone_service.py\", line 49, in invoke_module\n imp.load_module('__main__', mod, module, MOD_DESC)\n File \"/tmp/ansible_os_keystone_service_payload_EVkuxm/__main__.py\", line 194, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_EVkuxm/__main__.py\", line 150, in main\n File \"/tmp/ansible_os_keystone_service_payload_EVkuxm/ansible_os_keystone_service_payload.zip/ansible/module_utils/openstack.py\", line 126, in openstack_cloud_from_module\n File \"/usr/lib64/python2.7/distutils/version.py\", line 40, in __init__\n self.parse(vstring)\n File \"/usr/lib64/python2.7/distutils/version.py\", line 105, in parse\n match = self.version_re.match(vstring)\nTypeError: expected string or buffer\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1,
    "retries": 6
}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/711449

[Mark Goddard (mgoddard)]

Was finally able to reproduce:

virtualenv venv
venv/bin/pip install -c https://releases.openstack.org/constraints/upper/master 'ansible<2.9' openstacksdk
cat << EOF > repro.yml
---
- hosts: localhost
  gather_facts: no
  tasks:
    - os_keystone_service:
        auth:
          auth_url: "http://localhost:35357"
          domain_name: "default"
          password: "1zwBct8GtM44jb97UuM2l5vhcs85UzbA3snC3et3"
          project_name: "admin"
          user_domain_name: "default"
          username: "admin"
        name: foo
        service_type: bar
EOF

venv/bin/ansible-playbook repro.yml

PLAY [localhost] *******************************************************************************************************************************************************************************************

TASK [os_keystone_service] *********************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: expected string or buffer
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/home/mark/.ansible/tmp/ansible-tmp-1583417741.62-12087961222203/AnsiballZ_os_keystone_service.py\", line 114, in <module>\n _ansiballz_main()\n File \"/home/mark/.ansible/tmp/ansible-tmp-1583417741.62-12087961222203/AnsiballZ_os_keystone_service.py\", line 106, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/home/mark/.ansible/tmp/ansible-tmp-1583417741.62-12087961222203/AnsiballZ_os_keystone_service.py\", line 49, in invoke_module\n imp.load_module('__main__', mod, module, MOD_DESC)\n File \"/tmp/ansible_os_keystone_service_payload_WbPSEt/__main__.py\", line 194, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_WbPSEt/__main__.py\", line 150, in main\n File \"/tmp/ansible_os_keystone_service_payload_WbPSEt/ansible_os_keystone_service_payload.zip/ansible/module_utils/openstack.py\", line 126, in openstack_cloud_from_module\n File \"/usr/lib/python2.7/distutils/version.py\", line 40, in __init__\n self.parse(vstring)\n File \"/usr/lib/python2.7/distutils/version.py\", line 105, in parse\n match = self.version_re.match(vstring)\nTypeError: expected string or buffer\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

This works with ansible 2.8.8.

[Mark Goddard (mgoddard)]

Looks like it was caused by this patch: https://github.com/ansible/ansible/commit/e9a190e9c19cb4898c2f264be099f625f77a5f14.

It is a backport, but has some additional changes in module_utils/openstack.py which are the issue.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: master
Review: https://review.opendev.org/711449
Reason: Wrong fix.

[Mark Goddard (mgoddard)]

PR for above patch: https://github.com/ansible/ansible/pull/67577/files

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/711470

[Mark Goddard (mgoddard)]

Raised a bug in Ansible (https://github.com/ansible/ansible/issues/68042) and proposed a fix (https://github.com/ansible/ansible/pull/68043).

[Mark Goddard (mgoddard)]

Related kayobe bug: https://storyboard.openstack.org/#!/story/2007383

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711470
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=a755304795eb7bd43a26b7380aa2686151295c64
Submitter: Zuul
Branch: master

commit a755304795eb7bd43a26b7380aa2686151295c64
Author: Mark Goddard <email address hidden>
Date: Thu Mar 5 14:34:18 2020 +0000

    Multiple fixes to unblock CI

    This change includes a number of fixes to make CI functional.

    * Blacklist ansible 2.8.9 in kolla-toolbox

      Ansible 2.8.9 broke the OpenStack modules, see
      https://bugs.launchpad.net/kolla/+bug/1866181 for details.

      This causes us to fail with the following error:

        /usr/lib64/python2.7/distutils/version.py\", line 40, in __init__
        self.parse(vstring)
        File "/usr/lib64/python2.7/distutils/version.py", line 105, in parse
        match = self.version_re.match(vstring)\nTypeError: expected string or buffer

      This change works around the issue by blacklisting ansible 2.8.9 in
      the kolla-toolbox image.

    * Drop CentOS 7 CI jobs. There are multiple issues with CentOS 7 now,
      and we have CentOS 8 jobs in place so it's time to move on

    * pin requirements to commit
      ea42244642c3c091cb4f70917ae8d79af217b6af. This is required to avoid
      oslo.cache 2.1.0, which breaks us. See
      https://bugs.launchpad.net/kolla-ansible/+bug/1866008.

    Change-Id: Iebc39bd804d2c4f553d42c285549f50f4c8bc8be
    Closes-Bug: #1866181
    Related-Bug: #1866008

[Mark Goddard (mgoddard)]

The fix for ansible was released yesterday in ansible 2.8.10: https://groups.google.com/forum/#!msg/ansible-devel/Zz3cwmDlyOA/jEIkG6x2AAAJ