Stable branch CI fails when gathering facts - invalid syntax (fake_filesystem.py, line 1195)

[Mark Goddard]

On train and earlier stable branches, CI is failing in the following task:

2020-03-04 12:42:33.898606 | TASK [Record the running state of the environment as seen by the setup module]
2020-03-04 12:42:35.824197 | primary | ERROR! Unexpected Exception, this is probably a bug: invalid syntax (fake_filesystem.py, line 1195)
2020-03-04 12:42:35.979719 | primary | ERROR
2020-03-04 12:42:35.980054 | primary | {
2020-03-04 12:42:35.980172 | primary | "delta": "0:00:01.671291",
2020-03-04 12:42:35.980299 | primary | "end": "2020-03-04 12:42:35.872841",
2020-03-04 12:42:35.980409 | primary | "msg": "non-zero return code",
2020-03-04 12:42:35.980516 | primary | "rc": 250,
2020-03-04 12:42:35.980642 | primary | "start": "2020-03-04 12:42:34.201550"
2020-03-04 12:42:35.980746 | primary | }

On train this happens on CentOS 7 (not 8) and Debian jobs. These jobs use Ansible 2.6 or earlier, whereas Ubuntu and CentOS 8 use Ansible 2.8 (on train).

[Mark Goddard (mgoddard)]

The command that is executed during that task is:

ansible all -i {{ kolla_inventory_path }} -e ansible_user={{ ansible_user }} -m setup > /tmp/logs/ansible/initial-setup

Essentially, gathering facts.

Reproducing on a CentOS 7 box:

sudo yum -y install python-virtualenv gcc libffi-devel openssl-devel
virtualenv ansible-2.6
source ansible-2.6/bin/activate
pip install 'ansible<2.7' 'ara<1.0.0'
cat << EOF > ansible.cfg
[defaults]
callback_plugins = $(pwd)/ansible-2.6/lib/python2.7/site-packages/ara/plugins/callbacks
EOF
./ansible-2.6/bin/ansible localhost -m setup -e ansible_python_interpreter=$(which python) -vvvv

This fails:

ansible 2.6.20
  config file = /home/mark/ansible.cfg
  configured module search path = [u'/home/mark/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/mark/ansible-2.6/lib/python2.7/site-packages/ansible
  executable location = /home/mark/ansible-2.6/bin/ansible
  python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
Using /home/mark/ansible.cfg as config file
setting up inventory plugins
Parsed /etc/ansible/hosts inventory source with ini plugin
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

Loading callback plugin minimal of type stdout, v2.0 from /home/mark/ansible-2.6/lib/python2.7/site-packages/ansible/plugins/callback/minimal.pyc
ERROR! Unexpected Exception, this is probably a bug: invalid syntax (fake_filesystem.py, line 1195)
the full traceback was:

Traceback (most recent call last):
  File "/home/mark/ansible-2.6/bin/ansible", line 118, in <module>
    exit_code = cli.run()
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ansible/cli/adhoc.py", line 181, in run
    result = self._tqm.run(play)
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 231, in run
    self.load_callbacks()
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 193, in load_callbacks
    for callback_plugin in callback_loader.all(class_only=True):
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ansible/plugins/loader.py", line 489, in all
    module = self._load_module_source(name, path)
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ansible/plugins/loader.py", line 357, in _load_module_source
    module = imp.load_source(full_name, path, module_file)
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ara/plugins/callbacks/log_ara.py", line 29, in <module>
    from ara.webapp import create_app
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ara/webapp.py", line 39, in <module>
    from ara.utils import fast_count
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/ara/utils.py", line 24, in <module>
    import pyfakefs.fake_filesystem as fake_filesystem
  File "/home/mark/ansible-2.6/lib/python2.7/site-packages/pyfakefs/fake_filesystem.py", line 1195
    def utime(self, path, times=None, *, ns=None, follow_symlinks=True):
                                       ^
SyntaxError: invalid syntax

pyfakefs ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/711239

[Mark Goddard (mgoddard)]

Raised a bug in ARA: https://github.com/ansible-community/ara/issues/118.
Fix proposed to ARA: https://review.opendev.org/711240.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/711239
Reason: Merged into https://review.opendev.org/#/c/711214

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/711267

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/711268

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711268
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=b50d070ca8b065ab71b4e1ff76428439fb2ee1da
Submitter: Zuul
Branch: stable/rocky

commit b50d070ca8b065ab71b4e1ff76428439fb2ee1da
Author: Mark Goddard <email address hidden>
Date: Wed Mar 4 14:23:21 2020 +0000

    CI: Pin pyfakefs to <4 for Python 2

    Stable branches only.

    pyfakefs 4.0.0 dropped support for Python 2. It is used by the ARA
    callback plugin.

    Change-Id: Ifc56078f1152c89f5f947851f801799a692a346a
    Closes-Bug: #1866079

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711267
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=07c0b83bbdec57599c18303a38f9cc5152eaed67
Submitter: Zuul
Branch: stable/stein

commit 07c0b83bbdec57599c18303a38f9cc5152eaed67
Author: Mark Goddard <email address hidden>
Date: Wed Mar 4 14:23:21 2020 +0000

    CI: Pin pyfakefs to <4 for Python 2

    Stable branches only.

    pyfakefs 4.0.0 dropped support for Python 2. It is used by the ARA
    callback plugin.

    Change-Id: Ifc56078f1152c89f5f947851f801799a692a346a
    Closes-Bug: #1866079

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711214
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c61d0af87c1d8ec8eb6897de0cef426a0c74dfee
Submitter: Zuul
Branch: stable/train

commit c61d0af87c1d8ec8eb6897de0cef426a0c74dfee
Author: Mark Goddard <email address hidden>
Date: Wed Mar 4 12:15:54 2020 +0000

    Set openstack_release and kolla_source_version to train

    In stein and earlier releases, we set openstack_release to 'auto'. This
    would detect the version of kolla-ansible and use it as the container
    tag. On master, we then changed this to 'master', with the intention
    that it would be changed on each stable branch prior to release.
    However, as this was a new part of the release process, we forgot to do
    it for Train. This results in kolla-ansible using the 'master' images by
    default.

    The same is true for kolla_source_version, which affects the version of
    repos used in dev mode.

    This change sets openstack_release and kolla_source_version to 'train'.

    Also included in this patch is a fix for ARA on Python 2:

    pyfakefs 4.0.0 dropped support for Python 2. It is used by the ARA
    callback plugin.

    Change-Id: Ia0dc783929769c7c2dae5aaaad7d51c7de0e2b10
    Closes-Bug: #1866054
    Closes-Bug: #1866079

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.2.1 release.