devstack-plugin-ceph-tempest fails with "ImportError: cannot import name deploy"

[Matt Riedemann]

http://logs.openstack.org/55/569055/7/check/devstack-plugin-ceph-tempest/45135b5/controller/logs/devstacklog.txt.gz#_2018-11-13_14_03_58_828

2018-11-13 14:03:58.828 | + lib/glance:init_glance:304 : /usr/local/bin/glance-manage --config-file /etc/glance/glance-api.conf db_sync
2018-11-13 14:03:59.497 | Traceback (most recent call last):
2018-11-13 14:03:59.498 | File "/usr/local/bin/glance-manage", line 6, in <module>
2018-11-13 14:03:59.498 | from glance.cmd.manage import main
2018-11-13 14:03:59.498 | File "/opt/stack/glance/glance/cmd/manage.py", line 50, in <module>
2018-11-13 14:03:59.498 | from glance.common import config
2018-11-13 14:03:59.498 | File "/opt/stack/glance/glance/common/config.py", line 26, in <module>
2018-11-13 14:03:59.498 | from paste import deploy
2018-11-13 14:03:59.498 | ImportError: cannot import name deploy

Doesn't look like this is very new:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ImportError%3A%20cannot%20import%20name%20deploy%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22devstack-plugin-ceph-tempest%5C%22&from=10d

[Dr. Jens Harbott (j-harbott)]

So here's what I think is happening:

1. ceph gets installed from queens UCA and pulls in python-paste + python-pastedeploy pkgs.

http://logs.openstack.org/55/569055/7/check/devstack-plugin-ceph-tempest/45135b5/controller/logs/devstacklog.txt.gz#_2018-11-13_13_54_16_273

Versions are
python-paste 2.0.3+dfsg-4ubuntu1~cloud0
python-pastedeploy 1.5.2-1

2. Later keystone and its requirements are installed via pip, and it finds

http://logs.openstack.org/55/569055/7/check/devstack-plugin-ceph-tempest/45135b5/controller/logs/devstacklog.txt.gz#_2018-11-13_13_56_04_644

Requirement already satisfied: PasteDeploy===1.5.2 in /usr/lib/python2.7/dist-packages (from -c /opt/stack/requirements/upper-constraints.txt (line 20))
Collecting Paste===3.0.4 (from -c /opt/stack/requirements/upper-constraints.txt (line 19))

and then pip removes the distro-installed paste, replacing it with the newer version. But since it installs the new version into /usr/local/lib, it effectively shadows the system-install paste.deploy path.

This is exactly one of the issues that pip >= 10 wants to protect us from, but we are still pinning pip < 10 and thus exposed to this failure.

As a short-term workaround, doing a "pip uninstall PasteDeploy" and then "pip install PasteDeploy" solve the issue. But we really should continue to work on unpinning pip again, see https://review.openstack.org/561597

[Dr. Jens Harbott (j-harbott)]

So this seems to have been triggered by Paste==3.0.0 having been released about three weeks ago and then having made its way into upper-contraints about a week later here: https://review.openstack.org/#/c/614078/

We already have a similar workaround in place for the stable/queens branch running on Pike UCA, so just extending that for master again should work for now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/617985

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617985
Committed: https://git.openstack.org/cgit/openstack/devstack-plugin-ceph/commit/?id=39de6df04130cf2f221fb5ba2a9b5ff597de332a
Submitter: Zuul
Branch: master

commit 39de6df04130cf2f221fb5ba2a9b5ff597de332a
Author: Jens Harbott <email address hidden>
Date: Wed Nov 14 15:23:04 2018 +0000

    Fix installation of PasteDeploy from upstream pkgs (again)

    Similar to what happened some time ago for the stable/queens branch, we
    are now seeing issues again on master after there was a new Paste
    release being made a couple of weeks ago. So we need to perform
    the same workaround of forcing pip to install PasteDeploy over
    the distro version on the master branch.

    Change-Id: I3e98a08e78182bdf5ec09b30e31b98435ec29456
    Closes-Bug: 1803143