The python-requests package is pulled in by apt via dependency

[Jordan Callicoat]

The python-requests package is being pulled in from apt by lxc-templates.

If this happens after the install of the requests pip wheel, nova image-create and migrate (and potentially others) run into LP:1476770, where an image object with an empty properties dictionary is returned to glanceclient, which raises an AttributeError for key "id".

The lxc-templates package recommends cloud-image-utils, which in turn has a direct dependency on euca2ools, which has a direct dependency on python-requests.

root@87b035a9-compute1:~# apt-cache depends lxc-templates | fgrep cloud-image-utils
 |Recommends: cloud-image-utils

root@87b035a9-compute1:~# apt-cache depends cloud-image-utils | fgrep euca2ools
  Depends: euca2ools

root@87b035a9-compute1:~# apt-cache depends euca2ools | fgrep python-requests
  Depends: python-requests

We are using the apt module to install lxc-templates, which defaults to installing recommended packages unless install_recommends is set to no.

In the 10 series it is being installed here:

rpc_deployment/playbooks/setup/roles/lxc_common/tasks/lxc_install.yml

In the 11 series it is installed here:

playbooks/roles/lxc_hosts/defaults/main.yml (lxc_apt_packages variable)
playbooks/roles/lxc_hosts/tasks/lxc_install.yml

[Jesse Pretorius (jesse-pretorius)]

As a workaround to avoid this issue it would be best to execute administrative commands from the Utility container. This works around the issue while the upstream bug is resolved.

[Nolan Brubaker (nolan-brubaker)]

Was this bug report filed as a response to a client not working? Or was this a problem with some service misbehaving?

[Charles Farquhar (cfarquhar)]

Jordan can provide further details, but I believe this was filed in response to previously working environments having 'nova migrate' and 'nova image-create' functionality break for multiple customers. In order to restore functionality, the following commands were run on all of the hypervisors:

# apt-get remove python-requests -y ; pip install requests ; service nova-compute restart ; service neutron-linuxbridge-agent restart

However, the following packages were also removed as they have dependencies on python-requests:

  cloud-image-utils euca2ools python-requestbuilder python-requests
  ssh-import-id

I'm not sure what the impact of these packages being removed is.

[Nolan Brubaker (nolan-brubaker)]

What tag was this happening in? All 10.1.x?

[Nolan Brubaker (nolan-brubaker)]

Also are these commands running on a physical host, or within the utility containers?

[Jordan Callicoat (jcallicoat)]

We hit this bug in two new deployments and one upgraded environment.

The issue manifests with glanceclient on the compute hosts (the traceback in nova-compute.log is the same as in LP:1476770 ). The only work-around I've found is to remove python-requests via apt and install requests via pip on all the compute hosts (just running pip install --force-reinstall requests may work as well).

I will update with more information on tags, package versions and steps to reproduce.

[Jordan Callicoat (jcallicoat)]

Just ran into this in another environment.

root@<redacted>:~# cat /etc/rpc-release
DISTRIB_ID="RPC"
DISTRIB_RELEASE="10.1.11"
DISTRIB_CODENAME="wee-5ive"
DISTRIB_DESCRIPTION="Rackspace Private Cloud"

root@<redacted>:~# dpkg -l | fgrep python-requests
ii python-requests 2.2.1-1ubuntu0.3 all elegant and simple HTTP library for Python, built for human beings

root@<redacted>:~# pip freeze | fgrep requests
requests==2.2.1

I just checked another environment where we encountered the issue, and it was also 10.1.11 with the same package versions as above.

The upgraded environment where we also saw the issue is on 10.1.12 with the same package versions as above.

I was not able to reproduce in a lab running 10.1.3, nor in a lab running 11.0.0rc8, however they both have a different version of requests installed via pip than the python-requests package, whereas the environments above have the same version from pip and apt. Might have something to do with it?

v10.1.3 lab:

root@87b035a9-compute1:~# pip freeze | fgrep requests
requests==2.4.1

root@87b035a9-compute1:~# dpkg -l | fgrep python-requests
ii python-requests 2.2.1-1ubuntu0.3 all elegant and simple HTTP library for Python, built for human beings

v11.0.0rc8 lab:

root@1c30f32d-compute1:~# pip freeze | fgrep requests
requests==2.7.0

root@1c30f32d-compute1:~# dpkg -l | fgrep python-requests
ii python-requests 2.2.1-1ubuntu0.3 all elegant and simple HTTP library for Python, built for human beings

To reproduce on 10.1.11 / 10.1.12

1. Run "apt-get remove lxc-templates python-requests -y" on a compute host
2. Run "nova image-create" - works
3. Run "ansible-playbook -e @/etc/rpc_deploy/user_variables.yml lxc-hosts-setup.yml --limit <compute host>"
4. Run "nova image-create" - fails with traceback mentioned in other ticket

[Jordan Callicoat (jcallicoat)]

NB: The nova image-create command should be used with an instance that resides on the compute host you are testing on, if that was unclear! The error will be logged in /var/log/nova/nova-compute.log on the host where the instance lives.

[Jordan Callicoat (jcallicoat)]

Another new 10.1.11 deployment hit this issue today.

[Jesse Pretorius (jesse-pretorius)]

@Jordan - you provided feedback in IRC "that the issue is effecting nova-compute on all the hosts--when it tries to get image info from glance it gets back an empty Image object and raises an AttributeError". I'm trying to understand how it is that the lxc-templates templates package is being installed on the compute nodes?

[Jordan Callicoat (jcallicoat)]

@Jesse - Since there's an rsyslog container on compute hosts in the 10 series I assume it is pulling in the lxc-common role for the host, and the lxc_install task installs the lxc-templates package via apt. Where you unable to reproduce with the steps I gave?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/217628

[Jesse Pretorius (jesse-pretorius)]

@Jordan I've got a proposed fix which I'll test now. It seems that the version of novaclient which we're using is a little old - I'm changing the requirements and building wheels in the repo to test and see if the update fixes the issue.

[Jesse Pretorius (jesse-pretorius)]

Changing priority to high as this issue results in a dysfunctional environment.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: juno
Review: https://review.openstack.org/217628
Reason: Abandoning in favor of https://review.openstack.org/217098

[Jesse Pretorius (jesse-pretorius)]

@Jordan I've confirmed that the changes applied in https://review.openstack.org/217098 result in nova image-create working without any apt package changes. That review should be the final patch before we can release 10.1.13

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/217098
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=5bb99c8fbac479e74c93f954b48d316e7c1755d4
Submitter: Jenkins
Branch: juno

commit 5bb99c8fbac479e74c93f954b48d316e7c1755d4
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 26 12:23:41 2015 +0100

    Updated juno to include fix for CVE-2015-3241 - 26 Aug 2015

    Updates all repo SHAs to include:
     - [OSSA 2015-015] Nova instance migration process does not stop when
       instance is deleted (CVE-2015-3241)
     - Cannot rebuild a VM created from a Cinder volume backed by NetApp
       https://review.openstack.org/203253

    This patch removes all SHA specifications for the OpenStack clients
    so that any client requirement versions are determined solely from
    the requirements of the OpenStack service and Global requirements
    files.

    This resolves issues where the client versions we carry are
    incompatible with the services, resulting in unexpected failures.

    Tempest requirements are specifically ignored as the tempest role
    downloads its required clients directly from pip.

    Finally, due to the change in the openstack-client version used the
    heat domain/role/user tasks fail as they require the CLI to specify
    the user and project domain name/id. This patch includes the CLI
    changes to ensure that the deployment completes properly.

    Change-Id: I88e705d063575f55c32350ead25bc7cc0bcdec08
    Closes-Bug: #1489947
    Closes-Bug: #1488315
    Closes-Bug: #1400881

[Charles Farquhar (cfarquhar)]

Please note that we are still seeing this issue in 10.1.13.

[Jesse Pretorius (jesse-pretorius)]

The issue is still there and without some major refactoring will remain there - we'll try to find a suitable workaround that gives reliable results. Any suggestions are welcome.

[Bjoern (bjoern-t)]

My findings to the problem:

Pip should always install under /usr/local/lib and the debian packages only on the system libraries (/usr/lib)
When it comes down to this issue, the PYTHONPATH prefers /usr/lib before /usr/local and unfortunately for us,
the request library from Ubuntu is patched in a way not loading urllib3 over the the embedded .packages.urllib3 path.
And I think this is the more important issue since we installed urllib3 in version 1.12 and Ubuntu 1.7.1
Other than that, the requests library from Ubuntu and ours are the same with some minor HTTP proxy fixes.

[Ian Cordasco (icordasc)]

One of the biggest problems here is that we're (unnecessarily, as I understand it) installing oslo.vmware on baremetal on the compute hosts. That is the *only* package in all of openstack that actively requires urllib3 in Juno/Kilo. Since we keep calling Juno "Rackspace Technical Debt" and we know our customers, let's just stop installing oslo.vmware on those nodes. This will almost certainly solve the problems here.

[Jesse Pretorius (jesse-pretorius)]

Upstream discussion on this topic - the same issue is now affecting Kilo. https://bugs.launchpad.net/python-glanceclient/+bug/1476770

[Jesse Pretorius (jesse-pretorius)]

I'm removing myself as an assignee as I'm not able to work on this. I've also marked the Kilo branch as 'New' as I don't know if this is an issue on Kilo.

[Jesse Pretorius (jesse-pretorius)]

Setting Juno series target to "Won't Fix" as there will be no more adjustments going into the Juno branch unless they are Critical bugs.

[Kevin Carter (kevin-carter)]

This issue has been marked invalid because of our use of venvs which further isolates the host python from the OpenStack python bits.