grenade fails due to PyYAML conflict

[Michael Johnson]

When running grenade from stable/train to Ussuri, it fails when installing the new version of keystone (3.6) with:
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

This is due to using a newer version of pip (>10) and the fact that python3-yaml (3.12) is already installed by nova.

An example run log:
https://zuul.opendev.org/t/openstack/build/77b9004d463f472796638112c2cd6edc/log/logs/grenade.sh.txt.gz

Attempting to uninstall python3-yaml before the grenade run does not work around the issue.

This was seen on at least the designate and octavia grenade gate runs. Octavia has resorted to making grenade non-voting.

[Michael Johnson (johnsom)]

It looks like devstack is installing the python3-yaml that later conflicts with keystone:

2019-11-26 19:24:02.010499 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L265: local localrc_oldnew=old
2019-11-26 19:24:02.010834 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L266: local localrc_file=devstack.local.conf.base
2019-11-26 19:24:02.011127 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L267: local role=primary
2019-11-26 19:24:02.011456 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L273: local branch_for_matrix=stable/train
2019-11-26 19:24:02.011726 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L277: [[ -z '' ]]
2019-11-26 19:24:02.011969 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L278: rm -f devstack.local.conf.base
2019-11-26 19:24:02.014291 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L282: [[ ! -z '' ]]
2019-11-26 19:24:02.015056 | primary | ++ /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L286: which python3
2019-11-26 19:24:02.016667 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L286: PYTHON_PATH=/usr/bin/python3
2019-11-26 19:24:02.017340 | primary | ++ /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L287: basename /usr/bin/python3
2019-11-26 19:24:02.018730 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L287: PYTHON_NAME=python3
2019-11-26 19:24:02.018969 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L288: uses_debs
2019-11-26 19:24:02.019167 | primary | + /opt/stack/new/devstack-gate/functions.sh:uses_debs:L52: type apt-get
2019-11-26 19:24:02.019388 | primary | apt-get is /usr/bin/apt-get
2019-11-26 19:24:02.019620 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L289: dpkg -s python3-yaml
2019-11-26 19:24:02.045207 | primary | dpkg-query: package 'python3-yaml' is not installed and no information is available
2019-11-26 19:24:02.045409 | primary | Use dpkg --info (= dpkg-deb --info) to examine archive files,
2019-11-26 19:24:02.045537 | primary | and dpkg --contents (= dpkg-deb --contents) to list their contents.
2019-11-26 19:24:02.046222 | primary | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:setup_localrc:L290: apt_get_install python3-yaml
2019-11-26 19:24:02.046457 | primary | + /opt/stack/new/devstack-gate/functions.sh:apt_get_install:L63: [[ -z '' ]]
2019-11-26 19:24:02.046800 | primary | + /opt/stack/new/devstack-gate/functions.sh:apt_get_install:L64: timeout 300 sh -c 'while ! sudo apt-get update; do sleep 30; done'
2019-11-26 19:24:02.215036 | primary | Ign:1 http://mirror.ord.rax.opendev.org/ubuntu bionic InRelease
2019-11-26 19:24:02.215742 | primary | Ign:2 http://mirror.ord.rax.opendev.org/ubuntu bionic-updates InRelease
2019-11-26 19:24:02.216532 | primary | Ign:3 http://mirror.ord.rax.opendev.org/ubuntu bionic-backports InRelease
2019-11-26 19:24:02.218796 | primary | Ign:4 http://mirror.ord.rax.opendev.org/...

[Michael Johnson (johnsom)]

https://zuul.opendev.org/t/openstack/build/fff672698562463fb30ac21654e454e9/log/job-output.txt#2595

Example log

[Michael Johnson (johnsom)]

It appears to be the devstack-vm-gate.sh that is installing the python3-yaml package.

https://github.com/openstack/devstack-gate/blob/master/devstack-vm-gate.sh#L281

[Michael Johnson (johnsom)]

keystone test-requirements.txt is what is pulling in the 5.1.2:

2019-11-26 20:20:22.434 | + inc/python:setup_package:523 : pip_install -e /opt/stack/new/keystone
2019-11-26 20:20:22.461 | Check python version for : /opt/stack/new/keystone
2019-11-26 20:20:22.466 | Using 3.6 version to install /opt/stack/new/keystone based on default behavior
2019-11-26 20:20:23.348 | + inc/python:pip_install:274 : local install_test_reqs=
2019-11-26 20:20:23.351 | + inc/python:pip_install:275 : local test_req=/opt/stack/new/keystone/test-requirements.txt
2019-11-26 20:20:23.354 | + inc/python:pip_install:276 : [[ -e /opt/stack/new/keystone/test-requirements.txt ]]
2019-11-26 20:20:23.357 | + inc/python:pip_install:277 : install_test_reqs='-r /opt/stack/new/keystone/test-requirements.txt'
2019-11-26 20:20:23.359 | + inc/python:pip_install:284 : sudo -H LC_ALL=en_US.UTF-8 http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite /usr/local/bin/pip3.6 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/keystone/test-requirements.txt -e /opt/stack/new/keystone
2019-11-26 20:20:24.044 | Ignoring Django: markers 'python_version == "2.7"' don't match your environment
2019-11-26 20:20:24.046 | Ignoring Django: markers 'python_version == "3.4"' don't match your environment
2019-11-26 20:20:24.049 | Ignoring Django: markers 'python_version == "3.5"' don't match your environment
2019-11-26 20:20:24.054 | Ignoring Django: markers 'python_version == "3.7"' don't match your environment
2019-11-26 20:20:24.063 | Ignoring GitPython: markers 'python_version == "2.7"' don't match your environment
2019-11-26 20:20:24.065 | Ignoring GitPython: markers 'python_version == "3.4"' don't match your environment
2019-11-26 20:20:24.068 | Ignoring GitPython: markers 'python_version == "3.5"' don't match your environment
2019-11-26 20:20:24.073 | Ignoring GitPython: markers 'python_version == "3.7"' don't match your environment
2019-11-26 20:20:24.078 | Ignoring M2Crypto: markers 'python_version == "2.7"' don't match your environment
2019-11-26 20:20:24.105 | Ignoring SecretStorage: markers 'python_version == "2.7"' don't match your environment
2019-11-26 20:20:24.108 | Ignoring SecretStorage: markers 'python_version == "3.4"' don't match your environment
2019-11-26 20:20:24.110 | Ignoring SecretStorage: markers 'python_version == "3.5"' don't match your environment
2019-11-26 20:20:24.117 | Ignoring SecretStorage: markers 'python_version == "3.7"' don't match your environment
2019-11-26 20:20:24.120 | Ignoring Sphinx: markers 'python_version == "2.7"' don't match your environment
2019-11-26 20:20:24.123 | Ignoring Sphinx: markers 'python_version == "3.4"' don't match your environment
2019-11-26 20:20:24.125 | Ignoring Sphinx: markers 'python_version == "3.5"' don't match your environment
2019-11-26 20:20:24.130 | Ignoring Sphinx: markers 'python_version == "3.7"' don't match your environment
2019-11-26 20:20:24.232 | Ignoring asyncio: markers 'python_version == "3.4"' don't match your environment
2019-11-26 20:20:24.234 | Ignoring async...