[vagrant] openstack-ansible-lxc_hosts: The `lxc` module is not importable. Check the requirements."

[Markos Chandras]

Hi,

Running 'vagrant up' on the openstack-ansible-lxc_hosts (84ae0818f5e6f556830ec0e990a57fbaddaefd9d) results to: "The `lxc` module is not importable. Check the requirements."

So my theory is that this is because lxc-python2 package is installed in the '.tox/functional' virtualenv instead of the real root. I also noticed that there is a major difference between the 'vagrant' and the OpenStack CI tox execution

In the OpenStack CI jobs [1] we observe the following

2017-03-31 19:30:26.222231 | "cmd": "/usr/bin/pip2 install -U lxc-python2",

but in vagrant we see the following

==> default: "cmd": "/vagrant/.tox/functional/bin/pip2 install -U lxc-python2",

So we see that a different 'pip' is used even though both environments use 'tox -e functional'. This probably results to 'lxc-python2' being installed int a wrong place.

Please let me know if you need me to provide more information but it's very easy to reproduce it with a simple 'vagrant up'

[1] http://logs.openstack.org/42/452142/1/check/gate-openstack-ansible-lxc_hosts-ansible-func-centos-7/f74729a/console.html.gz

[Markos Chandras (hwoarang)]

So a bit more information

It turns out what 'tox' prepends /vagrant/.tox/install/bin in $PATH. The resulting PATH in the vagrant system is

/vagrant/.tox/install/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

This probably results in finding the pip2 from /vagrant/.tox/install/bin instead of using the one from the host according to this

https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/packaging/language/pip.py#L319

and this

https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/basic.py#L1914

So basically, the pip module is trying to locate the 'pip' to use by calling the get_bin_path() method and this method iterates over the $PATH variable until it finds a suitable 'pip'

Of course this does not explain why in OpenStack CI it does not pick the pip from the .tox/$env directory. My only theory so far is that .tox/$env/bin/ does not have pip in it. I am not sure why. I am afraid that we need to have a look on one of the actual CI hosts to find out what's happening...

[Markos Chandras (hwoarang)]

Furthermore, when ansible runs the following sys.path is used

['/tmp/ansible_01I8rP', '/tmp/ansible_01I8rP/ansible_modlib.zip', '/tmp/ansible_01I8rP/ansible_modlib.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

As a result, anything is .tox/$venv/python2.7 (ie lxc) can't be found and imported.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/457695

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/457695
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=0fd42e3dc0484f2fe76914d29b04c9518f66ad4d
Submitter: Jenkins
Branch: master

commit 0fd42e3dc0484f2fe76914d29b04c9518f66ad4d
Author: Markos Chandras <email address hidden>
Date: Tue Apr 18 12:03:26 2017 +0100

    tox.ini: Set PYTHONPATH when executing tox tests

    Ansible seems to override the default PYTHONPATH so set PYTHONPATH
    explicitely to point to the virtualenv location so that ansible modules
    can import the LXC module.

    Link: https://github.com/ansible/ansible/blob/devel/lib/ansible/executor/module_common.py#L154
    Related-Bug: 1682108
    Change-Id: Ia87fdc30e200bf81992d6cf1dc3f49c64e900c2d

[Markos Chandras (hwoarang)]

So thanks to Kevin Carter for his assistance I got access to one of the OpenStack CI nodes to understand why the 'wrong' pip is used on them. The main reason for that is that the tests are being executed by the 'jenkins' user. When this user starts the 'tox' process the .tox/$venv/bin path is added to PATH. However, later on, when ansible uses sudo to do its job, PATH is reset to whatever is defined for root. As such, the pip module no longer looks for a pip executable in .tox/$venv/bin.

There are a couple of things to fix apart from what the previous commit fixed already

The first thing is to fix the Vagrantfile to start the tests as a normal user instead of root to mimic what happens in the OpenStack CI.

The other thing would be to see whether we actually want to fix it in Ansible somehow.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/458420

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458420
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-tests/commit/?id=ffd2d5cf34fd349bb4f44bc736935faa7704f6e1
Submitter: Jenkins
Branch: master

commit ffd2d5cf34fd349bb4f44bc736935faa7704f6e1
Author: Markos Chandras <email address hidden>
Date: Thu Apr 20 09:57:20 2017 +0100

    Vagrantfile: Do not run the tox tests as root

    By default, Vagrant runs the provision script as root. However, in the
    OpenStack CI such tests are being executed as normal user so we observe
    different behaviors when we compare the tests results between these two
    platforms. As such, add 'privileged: false' to the Vagrantfile to
    make Vagrant provision the VM as normal user.

    Closes-Bug: 1682108
    Change-Id: Ia4887d90d77110a88030a1958f57eddce5c4b2c2