devstack-gate jobs fail because ansible is not installed properly

[Matt Riedemann]

e.g.:

http://logs.openstack.org/39/193139/1/gate/gate-grenade-dsvm/0086d24/console.html#_2015-09-17_14_32_11_368

We see:

./safe-devstack-vm-gate-wrap.sh: line 563: /tmp/ansible/bin/ansible: No such file or directory

Confirmed in infra:

(3:46:54 PM) jeblair: mriedem: i see failures to pip install ansible, so probably hpcloud deleting our mirror account.
(3:47:23 PM) fungi: yep, it subsided when we reconfigured nodepool to stop booting new nodes in hpcloud
(3:47:38 PM) fungi: in theory
(3:47:57 PM) fungi: the timing on that log certainly looks like it's from the timeframe where it was still broken anyway

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiL3RtcC9hbnNpYmxlL2Jpbi9hbnNpYmxlOiBObyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5XCIgQU5EIGJ1aWxkX25vZGU6KmhwY2xvdWQqIEFORCB0YWdzOlwiY29uc29sZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDQyNTIyOTU1MTQyLCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

[Matt Riedemann (mriedem)]

e-r query: https://review.openstack.org/#/c/224893/

[Matt Riedemann (mriedem)]

This only hits on OVH now:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%2Ftmp%2Fansible%2Fbin%2Fansible%3A%20No%20such%20file%20or%20directory%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1&from=7d

[Pavlo Shchelokovskyy (pshchelo)]

I also seem to have similar issues on py3 unit tests jobs (was running on rax?)

http://logs.openstack.org/24/469624/2/check/gate-ironic-python35/9a19d98/console.html#_2017-06-01_06_46_58_000808

it seems there's some failure due to six is not available in the environment when setuptools tries to use it, which is also present in the logs of jobs described above (ansible was not installed due to the same failure - http://logs.openstack.org/24/469624/2/check/gate-tempest-dsvm-ironic-pxe_ipmitool-postgres-ubuntu-xenial-nv/ba57cc8/console.html#_2017-06-01_06_45_13_483721

and a new release of setuptools 36 was just pushed today (June 1, 2017). coincidence? :)

[Pavlo Shchelokovskyy (pshchelo)]

I also see such error in logs

Complete output from command python setup.py egg_info:
  Traceback (most recent call last):
    File "/home/jenkins/workspace/gate-ironic-python35/.tox/py35/lib/python3.5/site-packages/setuptools/dist.py", line 418, in fetch_build_egg
      cmd = self._egg_fetcher
  AttributeError: 'Distribution' object has no attribute '_egg_fetcher'

http://logs.openstack.org/24/469624/2/check/gate-ironic-python35/9a19d98/console.html#_2017-06-01_06_46_58_647784

[Pavlo Shchelokovskyy (pshchelo)]

oops, this is a very old bug...

[YAMAMOTO Takashi (yamamoto)]

see the following for the issue we're suffering from right now.
https://bugs.launchpad.net/devstack/+bug/1694904
https://github.com/pypa/setuptools/issues/1042

[Clark Boylan (cboylan)]

Updated the title to specifically mention devstack-gate as it is the tool relying on ansible being installed properly. Also, remove specific mention of the cloud involved as this issue can happen on any cloud when setuptools is not working. Hopefully this makes the general issue and how to tackle it more clear.