Mitaka: libvirt-python package not consistently build in repo

[Bjoern]

I was installing OSA 13.3.3 on two different cluster and we noticed difference around libvirt-python inside the venv:

Working :

libvirt_python-1.3.2-cp27-none-linux_x86_64.whl -> /var/www/repo/pools/libvirt_python/libvirt_python-1.3.2-cp27-none-linux_x86_64.whl

Not working :

libvirt_python-1.3.2-cp27-cp27mu-linux_x86_64.whl -> /var/www/repo/pools/libvirt_python/libvirt_python-1.3.2-cp27-cp27mu-linux_x86_64.whl

resulting in:

 >>> import libvirtmod
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 ImportError: /usr/lib/libvirt.so.0: version `LIBVIRT_1.2.16' not found (required by /openstack/venvs/nova-13.3.3/lib/python2.7/site-packages/libvirtmod.so)

The issue was fixed after I just copied the libvirt_python-1.3.2-cp27-none-linux_x86_64.whl wheel from a working deployment and installed it.
Personally I don't know why we not just copying /usr/lib/python2.7/dist-packages/libvirtmod* into the venv, that way we would make sure that the libvirt version and libvirt-python do match.

All the other libraries like libvirt/qemu where exactly the same so it is not known why the repo build process builds entirely different wheels even if all requirements.txt match

[Jesse Pretorius (jesse-pretorius)]

The difference in the wheel name is due to a different compiler being used when compiling the wheel. Similar issues have been seen where the compute node where libvirt is installed has a different version installed to the repo server where the wheel is compiled. This may also be an issue where a suitable wheel is found on an extra index provided to the repo build process, but there's a mismatch in the packages installed between the compute host and the host where the wheel was compiled.

In order to determine the real root cause here, please can you supply the package list for the compute host and the repo server for both installs. Also please provide The following var values for both environments:

repo_build_pip_default_index
repo_build_pip_extra_index
repo_build_pip_extra_indexes

In Newton we added the repo_build_pip_no_binary variable in order to force the repo build process to always build packages in the list provided from source (never use wheels from any indexes). This covers the situation where an extra index is used but may not have the right binaries installed. If the environment has a homogenous build (same package versions on the repo container and the compute hosts) then this is perfectly good, and Newton already ensures that python-libvirt is set to always build from source.

If it is desired to allow the environment to have non-homogenous builds or inconsistent package versions between the repo server and the compute hosts then it may be better to implement a link from the nova venv into the system package for the python-libvirt system package in a similar way to how we do the ceph libs. I do think that it seems like a good operations practice to update the libvirt package at the same time as updating the nova venv though, so the linking should not be required.

[Bjoern (bjoern-t)]

From what I gathered, the environment look pretty similar.
Still not sure why no libvirt is installed on the "good" repo container unlike the new (bad) one.

good (repo/infra) host:
-----------------------
repo_build_pip_extra_indexes:
  - "https://rpc-repo.rackspace.com/pools"

No libvirt installed on repo container or infra node

bad (repo/infra) host:
----------------------
repo_build_pip_extra_indexes:
  - "https://rpc-repo.rackspace.com/pools"

libvirt-bin:amd64 (1.2.2-0ubuntu13.1.17) inside the repo container
libvirt-dev:amd64 (1.2.2-0ubuntu13.1.17) inside the repo container

good compute host:
-----------------
libvirt-python (1.3.2) inside nova-13.3.3 venv
libvirt0:amd64 (1.2.2-0ubuntu13.1.17)
libvirt-bin:amd64 (1.2.2-0ubuntu13.1.17
python-libvirt:amd64 (1.2.2-0ubuntu2)

bad compute host:
-----------------
libvirt-python (1.3.2) inside nova-13.3.3 venv
libvirt0:amd64 (1.2.2-0ubuntu13.1.17)
libvirt-bin:amd64 (1.2.2-0ubuntu13.1.17
python-libvirt:amd64 (1.2.2-0ubuntu2)

[Jean-Philippe Evrard (jean-philippe-evrard)]

Could you give us more history about this?

It doesn't look right that libvirt-bin is installed in the repo-container.
It doesn't look to NOT have libvirt-dev either.

I can't reproduce this myself. Clean repo-containers are not in the same state as yours.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/388027

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/388198
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=22c55a72b75c17df84529e09aaa51be9909c5b15
Submitter: Jenkins
Branch: stable/mitaka

commit 22c55a72b75c17df84529e09aaa51be9909c5b15
Author: Jesse Pretorius <email address hidden>
Date: Mon Jul 18 11:56:25 2016 +0100

    Add --no-binary option to wheel compile task

    There are situations where you may wish to ensure that a package
    is always built from source, regardless of whether a pre-compiled
    wheel is available on one of the indexes.

    This patch adds the ability to set a list of packages which must
    always be compiled from source.

    Combined backport of:
    - https://review.openstack.org/343573
    - https://review.openstack.org/388197

    Partial-Bug: #1625859
    Change-Id: I2f47d94c58503514a3f9a0a62081cbc1ca53613b
    (cherry picked from commit c06679ccc3568134fb7c0ab5068c26ecc1834d23)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/388027
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=faf78e457ee2be33fa275c4622067b9220fae903
Submitter: Jenkins
Branch: master

commit faf78e457ee2be33fa275c4622067b9220fae903
Author: Jesse Pretorius <email address hidden>
Date: Tue Oct 18 14:43:42 2016 +0100

    Work around bad libvirt-python wheel

    The following package must always build from source:

    - libvirt-python:
      A pre-built wheel can be missing libvirt capabilities
      from the installed version of libvirt-bin, leading to
      nova-compute failing to start.

    The pycparser upper constraint override is removed as
    the patch to make it compatible with upper constraints
    has merged.

    Closes-Bug: #1625859
    Change-Id: I3197f06839c9705574810ab616dff51924435da5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/388934

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/389064

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/389064
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=02701e3bf8370c15132e542c0fdec34c0dd31d60
Submitter: Jenkins
Branch: stable/mitaka

commit 02701e3bf8370c15132e542c0fdec34c0dd31d60
Author: Jesse Pretorius <email address hidden>
Date: Thu Oct 20 09:38:21 2016 +0100

    Work around bad libvirt-python wheel

    The following package must always build from source:

    - libvirt-python:
      A pre-built wheel can be missing libvirt capabilities
      from the installed version of libvirt-bin, leading to
      nova-compute failing to start.

    Note that this is a re-implementation of
    https://review.openstack.org/388027

    The re-implementation is necessary as there are no
    group_vars scoped to the repo servers. If these vars
    are added to group_vars/all then the compute nodes
    would also install from source every time, whereas
    it only needs to be done in the repo server's wheel
    build process.

    Closes-Bug: #1625859
    Change-Id: I3197f06839c9705574810ab616dff51924435da5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/388934
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=0b351dc62a4dd46f6fc0d2bba7861864675f60f8
Submitter: Jenkins
Branch: stable/newton

commit 0b351dc62a4dd46f6fc0d2bba7861864675f60f8
Author: Jesse Pretorius <email address hidden>
Date: Tue Oct 18 14:43:42 2016 +0100

    Work around bad libvirt-python wheel

    The following package must always build from source:

    - libvirt-python:
      A pre-built wheel can be missing libvirt capabilities
      from the installed version of libvirt-bin, leading to
      nova-compute failing to start.

    The pycparser upper constraint override is removed as
    the patch to make it compatible with upper constraints
    has merged.

    Closes-Bug: #1625859
    Change-Id: I3197f06839c9705574810ab616dff51924435da5
    (cherry picked from commit faf78e457ee2be33fa275c4622067b9220fae903)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 13.3.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 14.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 15.0.0.0b1 development milestone.