python-psutil dependency error on ubuntu 16.04.4

[ultranal]

during the installation process, pip would report dependency error when installing glance:
"Cannot uninstall ‘psutil’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall."
The error is reported because the package "psutil" is not installed by pip but apt, and the default version of python-psutil in ubuntu repository is 3.4.2-1 while the minimalist version glance needed is 5.4.3. And the package "python-psutil" itself was installed at the previous period of installation.

The solution is also obvious: just delete python-psutil from depended apt package list and let pip install and manage it.

[Jeremy Stanley (fungi)]

Corresponding issue for pip upstream: https://github.com/pypa/pip/issues/4805

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561427
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=f99d1771ba1882dfbb69186212a197edae3ef02c
Submitter: Zuul
Branch: master

commit f99d1771ba1882dfbb69186212a197edae3ef02c
Author: Hongbin Lu <email address hidden>
Date: Sat Apr 14 19:33:15 2018 +0000

    Do not use pip 10 or higher

    It looks pip 10 failed the uninstallation of distutils installed
    packages. This patch temporarily cap the version of pip to work-around.

    Closes-Bug: #1763966
    Change-Id: I8bf80efc04883cd754c19bea0303064080112c6e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/561471

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/561526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561471
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=6676f25a27d1c76fe21d5056760acff1513a2415
Submitter: Zuul
Branch: stable/queens

commit 6676f25a27d1c76fe21d5056760acff1513a2415
Author: Hongbin Lu <email address hidden>
Date: Sat Apr 14 19:33:15 2018 +0000

    Do not use pip 10 or higher

    It looks pip 10 failed the uninstallation of distutils installed
    packages. This patch temporarily cap the version of pip to work-around.

    Closes-Bug: #1763966
    Change-Id: I8bf80efc04883cd754c19bea0303064080112c6e
    (cherry picked from commit f99d1771ba1882dfbb69186212a197edae3ef02c)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/561594

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561526
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=8da4061966b8daabaf1ea118c1551e85985287ef
Submitter: Zuul
Branch: stable/pike

commit 8da4061966b8daabaf1ea118c1551e85985287ef
Author: Hongbin Lu <email address hidden>
Date: Sat Apr 14 19:33:15 2018 +0000

    Do not use pip 10 or higher

    It looks pip 10 failed the uninstallation of distutils installed
    packages. This patch temporarily cap the version of pip to work-around.

    Closes-Bug: #1763966
    Change-Id: I8bf80efc04883cd754c19bea0303064080112c6e
    (cherry picked from commit f99d1771ba1882dfbb69186212a197edae3ef02c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561594
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=0c980c6f8162e04c69ecfa46119e0d814963a764
Submitter: Zuul
Branch: stable/ocata

commit 0c980c6f8162e04c69ecfa46119e0d814963a764
Author: Hongbin Lu <email address hidden>
Date: Sat Apr 14 19:33:15 2018 +0000

    Do not use pip 10 or higher

    It looks pip 10 failed the uninstallation of distutils installed
    packages. This patch temporarily cap the version of pip to work-around.

    Closes-Bug: #1763966
    Change-Id: I8bf80efc04883cd754c19bea0303064080112c6e
    (cherry picked from commit f99d1771ba1882dfbb69186212a197edae3ef02c)

[Alexander Brand (alexbrand)]

I believe I am running into this issue when trying to stand up devstack from the stable/ocata tag:

```
  Found existing installation: Jinja2 2.8
    Uninstalling Jinja2-2.8:
      Successfully uninstalled Jinja2-2.8
  Found existing installation: PyYAML 3.11
Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Unmount /tmp/dib_build.7CPS0OfN/mnt/var/cache/apt/archives
Unmount /tmp/dib_build.7CPS0OfN/mnt/tmp/pip
Unmount /tmp/dib_build.7CPS0OfN/mnt/tmp/in_target.d
Unmount /tmp/dib_build.7CPS0OfN/mnt/tmp/ccache
Unmount /tmp/dib_build.7CPS0OfN/mnt/sys
Unmount /tmp/dib_build.7CPS0OfN/mnt/proc
Unmount /tmp/dib_build.7CPS0OfN/mnt/dev/pts
Unmount /tmp/dib_build.7CPS0OfN/mnt/dev
+/opt/stack/octavia/devstack/plugin.sh:build_octavia_worker_image:1 exit_trap
```

[Dr. Jens Harbott (j-harbott)]

@Alexander: While this looks similar, we need to treat each package that leads to conflicts seperately, please open a new bug report for that. Also, from the context of that log, that error seems to happen within octavia image creation, so that would be a bug for octavia and not devstack.

[Arjen de Rijke (arjenderijke)]

It appears that this "fix" might now be creating a new problem. While two days ago my devstack installation (script) worked fine, now it ends with this error:

+stack.sh:main:802 install_infra
+lib/infra:install_infra:31 local PIP_VIRTUAL_ENV=/opt/stack/requirements/.venv
+lib/infra:install_infra:32 '[' '!' -d /opt/stack/requirements/.venv ']'
+lib/infra:install_infra:32 virtualenv /opt/stack/requirements/.venv
New python executable in /opt/stack/requirements/.venv/bin/python
Complete output from command /opt/stack/requirements/.venv/bin/python -m pip config list:
ERROR: unknown command "config"
----------------------------------------
Traceback (most recent call last):
  File "/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 793, in main
    symlink=options.symlink,
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1087, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 935, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 964, in _install_wheel_with_search_dir
    config = _pip_config(py_executable, python_path)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1038, in _pip_config
    remove_from_env=["PIP_VERBOSE", "PIP_QUIET"],
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 886, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command /opt/stack/requirements/.venv/bin/python -m pip config list failed with error code 1

In the earlier logging you can see that pip is downgraded from version 19.0.1 to 9.0.3, the older version does not support the "config" command. So for some reason the "wrong" version ends up in the venv. If you remove the "fix", the old problem return, so that is not an option. As far as i can tell i did not change anything. What might be related is that on January 22nd pip version 19 was released. But stack.sh does too many things around pip to be certain what is going wrong. And it also uses "https://bootstrap.pypa.io/get-pip.py", to get a new pip version, so it is difficult to see which pip version is used when.

I run devstack on a centos-7 (virtual) machine. But i don't think the distro is relevant. Apart from the pip 19 release i don't see any obvious reasons why this suddenly stopped working

[Matthew Thode (prometheanfire)]

it seems like this will continue to be a problem until system python packages are not depended upon. virtualenv can be told to not use system python libs. I don't see pip ever willing to remove files not tracked by itself (so the old behaviour is not likely to return). It's time to move on and adapt to the new pip.

Eventually something will require a new version of pip (as more of a hard requirement and something that won't be backed out like https://github.com/pypa/virtualenv/issues/1301 was). This will cause devstack to 'hard lock' until it is updated to use new pip. Better to fix this now then let it surprise us.