Baremetal role installs python docker package on host via pip

[Mark Goddard]

The baremetal role installs the docker python package directly to the system python packages via pip [1]. This is necessary for interacting with the docker engine via the kolla_docker ansible module. However, using pip to install packages directly to the host can have negative consequences:

* It will overwrite any existing packages installed via a system package manager (yum, apt) or pip with new versions.
* The new version may not be compatible with other existing packages that depend on it.
* A system package update will overwrite the version installed by the baremetal role.

We hit the last issue recently, following a yum nightly update which installed a new version of python-urllib3 (1.10.2), which was incompatible with the version of requests (2.18.4) installed by the baremetal role as a dependency of the docker package. This broke the kolla_docker module with this output:

fatal: [MY_NODE]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Shared connection to MY_IP closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_12vatz/ansible_module_kolla_container_facts.py\", line 52, in <module>\r\n import docker\r\n File \"/usr/lib/python2.7/site-packages/docker/__init__.py\", line 2, in <module>\r\n from .api import APIClient\r\n File \"/usr/lib/python2.7/site-packages/docker/api/__init__.py\", line 2, in <module>\r\n from .client import APIClient\r\n File \"/usr/lib/python2.7/site-packages/docker/api/client.py\", line 6, in <module>\r\n import requests\r\n File \"/usr/lib/python2.7/site-packages/requests/__init__.py\", line 43, in <module>\r\n import urllib3\r\n File \"/usr/lib/python2.7/site-packages/urllib3/__init__.py\", line 10, in <module>\r\n from .connectionpool import (\r\n File \"/usr/lib/python2.7/site-packages/urllib3/connectionpool.py\", line 31, in <module>\r\n from .connection import (\r\n File \"/usr/lib/python2.7/site-packages/urllib3/connection.py\", line 45, in <module>\r\n from .util.ssl_ import (\r\n File \"/usr/lib/python2.7/site-packages/urllib3/util/__init__.py\", line 4, in <module>\r\n from .request import make_headers\r\n File \"/usr/lib/python2.7/site-packages/urllib3/util/request.py\", line 5, in <module>\r\n from ..exceptions import UnrewindableBodyError\r\nImportError: cannot import name UnrewindableBodyError\r\n", "msg": "MODULE FAILURE"}

The following workaround allowed kolla_docker to work:

pip uninstall requests
pip uninstall urllib3
yum remove python-requests python-urllib3
yum install python-requests python-urllib3

I suspect that only requests had any effect.

There's a little more context in the original kayobe issue [2].

Ideally, kolla-ansible would support installing the docker python package in a virtualenv, and using it when required.

There may be issues with always using a virtualenv on the remote hosts, for example the yum python package cannot be installed via pip. This would need investigation.

[1] https://github.com/openstack/kolla-ansible/blob/stable/pike/ansible/roles/baremetal/tasks/install.yml#L55
[2] https://github.com/stackhpc/kayobe/issues/63

[Mark Goddard (mgoddard)]

I have successfully tested the use of a virtualenv on the remote target hosts to work around this issue. This can be achieved by setting a host variable, ansible_python_interpreter, to a python interpreter installed in a virtualenv. Because kolla-ansible uses some python modules which are not available via PyPI, such as apt, yum, and selinux, it is typically necessary to create the virtualenv with --system-site-packages.

Ideally some support is required for installing this virtualenv, and the logical place for that is the kolla-ansible bootstrap-servers command and the baremetal role. I will propose a change which I used to test this.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/527093

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/527093
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=69979efc2e75dc4ab8e8e41a7136afdb64df678d
Submitter: Zuul
Branch: master

commit 69979efc2e75dc4ab8e8e41a7136afdb64df678d
Author: Mark Goddard <email address hidden>
Date: Thu Dec 7 11:44:05 2017 +0000

    Support virtualenv installation in baremetal role

    Installing python packages directly to the system site-packages can
    cause various problems, in particular when pip overwrites a system
    package. Python virtualenvs are one solution to this issue, as they
    allow python packages to be installed in an isolated environment.

    This change adds support to the baremetal role for installing python
    dependencies in a virtualenv. Typically we will need to enable use of
    system site-packages from within this virtualenv, to support the use of
    modules such as yum, apt, and selinux, which are not available on PyPI.

    The path to the virtualenv is configured via the 'virtualenv' variable,
    and access to site-packages is controlled via
    'virtualenv_site_packages'.

    When executing other kolla-ansible commands, the variable
    'ansible_python_interpreter' should be set to the python interpreter
    installed in 'virtualenv'. Note that this variable cannot be templated.

    Change-Id: I0741923065246f9c5b168059fcd66504f2753c41
    Related-Bug: #1731026

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/577382

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577382
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ddd9d6ae6d1ab0d6fed100343b090d023878a19b
Submitter: Zuul
Branch: stable/queens

commit ddd9d6ae6d1ab0d6fed100343b090d023878a19b
Author: Mark Goddard <email address hidden>
Date: Thu Dec 7 11:44:05 2017 +0000

    Support virtualenv installation in baremetal role

    Installing python packages directly to the system site-packages can
    cause various problems, in particular when pip overwrites a system
    package. Python virtualenvs are one solution to this issue, as they
    allow python packages to be installed in an isolated environment.

    This change adds support to the baremetal role for installing python
    dependencies in a virtualenv. Typically we will need to enable use of
    system site-packages from within this virtualenv, to support the use of
    modules such as yum, apt, and selinux, which are not available on PyPI.

    The path to the virtualenv is configured via the 'virtualenv' variable,
    and access to site-packages is controlled via
    'virtualenv_site_packages'.

    When executing other kolla-ansible commands, the variable
    'ansible_python_interpreter' should be set to the python interpreter
    installed in 'virtualenv'. Note that this variable cannot be templated.

    Change-Id: I0741923065246f9c5b168059fcd66504f2753c41
    Related-Bug: #1731026
    (cherry picked from commit 69979efc2e75dc4ab8e8e41a7136afdb64df678d)