tox fails with "Detected a distutils installed project ('argparse')"

[Ivan]

during env setup

```
...
py27 develop-inst: /home/jenkins/workspace/verify-fuel-web/nailgun
ERROR: invocation failed, logfile: /home/jenkins/workspace/verify-fuel-web/nailgun/.tox/py27/log/py27-2.log
ERROR: actionid=py27
...
```

pip tries to upgrde `argparse` but can't perform that operation

```
...
Collecting argparse (from oslo.config>=2.3.0->nailgun==9.0.0)
  Downloading argparse-1.4.0-py2.py3-none-any.whl
...
Installing collected packages: nailgun, argparse
  Running setup.py develop for nailgun
  Found existing installation: argparse 1.2.1
Detected a distutils installed project ('argparse') which we cannot uninstall. The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall.

...
```

[Ivan (iremizov)]

there is few possible solutions

- use `tox` for these tests. looks like it can handle such issues

- update `virtualenv` on fuel-ci slaves. It's possible that after updating it will use latest pip with latest argparse

[Ivan (iremizov)]

I've started job to check if redeployment helps on `ci-slave08.fuel-infra.org`

https://ci.fuel-infra.org/job/verify-fuel-web/8287/console

Bug is still there.

[Ivan (iremizov)]

jenkins@ci-slave08:~$ pip list
apt-xapian-index (0.45)
argparse (1.2.1)
chardet (2.0.1)
cloud-sptheme (1.6)
colorama (0.2.5)
DecoratorTools (1.8)
docutils (0.11)
flake8 (2.1.0)
gitdb (0.6.4)
GitPython (0.3.6)
gyp (0.1)
html5lib (0.999)
iniparse (0.4)
Jinja2 (2.7.2)
jsonschema (2.3.0)
libvirt-python (1.2.2)
lxml (3.3.3)
MarkupSafe (0.18)
mccabe (0.2.1)
mock (1.0.1)
networkx (1.8.1)
numpy (1.8.2)
pbr (0.7.0)
pdfrw (0.1)
pep8 (1.4.6)
Pillow (2.3.0)
pip (1.5.4)
psycopg2 (2.5.1)
py (1.4.20)
pyasn1 (0.1.8)
pycurl (7.19.3)
pyflakes (0.8.1)
Pygments (1.6)
pyliblzma (0.5.3)
pyOpenSSL (0.13)
pysqlite (1.0.1)
pytest (2.5.1)
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/list.py", line 80, in run
    self.run_listing(options)
  File "/usr/lib/python2.7/dist-packages/pip/commands/list.py", line 142, in run_listing
    self.output_package_listing(installed_packages)
  File "/usr/lib/python2.7/dist-packages/pip/commands/list.py", line 151, in output_package_listing
    if dist_is_editable(dist):
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 366, in dist_is_editable
    req = FrozenRequirement.from_dist(dist, [])
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist
    assert len(specs) == 1 and specs[0][0] == '=='
AssertionError

Storing debug log for failure in /home/jenkins/.pip/pip.log
jenkins@ci-slave08:~$

[Bartłomiej Piotrowski (bpiotrowski)]

To shed some light on the issue… Recently pip 8.0 was released which has problems with distutils based projects.

https://github.com/pypa/pip/issues/3384 – upstream report.

We could just put pip<8 in requirements to fix this.

[Ivan (iremizov)]

jenkins@ci-slave08:~$ pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)
jenkins@ci-slave08:~$

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/270830

[Aleksandra Fedorova (bookwar)]

https://review.openstack.org/#/q/topic:bug/1536188

We can pin pip version or disable updates in tox environments

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270830
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=767b44a0ef148dbbd41681f162a567d4f6c8f410
Submitter: Jenkins
Branch: master

commit 767b44a0ef148dbbd41681f162a567d4f6c8f410
Author: Aleksandra Fedorova <email address hidden>
Date: Thu Jan 21 17:32:56 2016 +0300

    Limit pip version for argparse

    pip 8.0 can not overinstall new argparse modules.

    Change-Id: I0240ec979a2de4d0e148919b36fece63dd22528a
    Related-Bug: #1536188

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/270901

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/270914

[Matthew Mosesohn (raytrac3r)]

I have deployment failed on custom 9.0 iso #943 in nailgun container:
2016-01-21 17:19:11 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_syncdb]/returns (notice): pkg_resources.DistributionNotFound: The 'pip<8.0' distribution was not found and is required by nailgun

You simply cannot add a python dependency to requirements.txt without adding it to the RPM spec. RPM is ignorant of what goes in there, and this breaks deployment. Please correct the RPM spec.

[Matthew Mosesohn (raytrac3r)]

Full nailgun puppet logs for #12: http://paste.openstack.org/show/ME1kGpILOURMmW93XS9L/

[Aleksandra Fedorova (bookwar)]

pip is not needed for nailgun to work, thus we shouldn't add it in rpm spec. So let's move it from requirements.txt to test-requirements.txt: https://review.openstack.org/#/c/271060/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/271193

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/271202

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andriy Popovych (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/270901
Reason: not relevant anymore

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oleg Gelbukh (<email address hidden>) on branch: master
Review: https://review.openstack.org/270914
Reason: Pip was fixed in 8.0.2, so no need to pin the version.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandra Fedorova (<email address hidden>) on branch: master
Review: https://review.openstack.org/271060
Reason: Obsoleted by https://review.openstack.org/#/c/271060/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andriy Popovych (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/271193
Reason: not relevant anymore

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/271202
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=73a318f6560d086610d13150a1fef5b3db468193
Submitter: Jenkins
Branch: master

commit 73a318f6560d086610d13150a1fef5b3db468193
Author: Evgeniy L <email address hidden>
Date: Fri Jan 22 12:31:56 2016 +0300

    Remove pinning of pip package

    It has been fixed in pip upstream
    https://github.com/pypa/pip/blob/8.0.2/CHANGES.txt#L11-L13

    Change-Id: If2cb933e0e000c776b4d4612ec559ab7185489d4
    Related-bug: #1536188

[Roman Prykhodchenko (romcheg)]

Folks, as far as I understand this is not an issue anymore?

[Aleksandra Fedorova (bookwar)]

pip update fixed the issue

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandra Fedorova (<email address hidden>) on branch: master
Review: https://review.openstack.org/270817
Reason: Not needed anymore