In mixed OS environments, the libvirt-python wheel does not work

[Jesse Pretorius]

The libvirt-python wheel has to be built with the same binary as will be used on the hosts that load it, otherwise the nova service will fail to start due to a mismatch of features between the library and the binary.

This presents a problem in mixed OS environments where it is not possible to build a wheel for each OS and deploy them selectively. Python's wheel naming doesn't allow for differences between OS's, only CPU architectures.

We have two options as far as I can see:

1. As discussed in https://bugs.launchpad.net/openstack-ansible/+bug/1637509 we could make the venv build process build a venv per arch and per OS in the environment. The caveat would have to be that at least one control plane host must be on each arch and on each OS version.

2. We could use the same mechanism we use for Ceph python libs and opt to symlink the python libs provided by the distro package into the venv. See
https://github.com/openstack/openstack-ansible-ceph_client/blob/d2ab0cee16ee1c3073a5e5f252c56e7b99dea7b6/tasks/ceph_install_python_libs.yml for reference.

Option 1 is complex. Option 2 is simple. I think I prefer option 2, but am open to alternative suggestions.

[Logan V (loganv)]

Option 2 is great if it works. But I am concerned that building on a repo container with uca/newton xenial libvirt installed, then installing the wheel on Trusty and linking the trusty uca/mitaka libs into the venv will possibly break the wheel.

[Jesse Pretorius (jesse-pretorius)]

@Logan If we go the linking route, then we'd remove the wheel build entirely.

Note:
Currently trusty-updates/mitaka on UCA has libvirt-bin 1.3.1-1ubuntu10.5~cloud0
and xenial has 1.3.1-1ubuntu10.5, so they do match in versions.

However, CentOS may be a different story. Also, UCA for Trusty is only maintained for the Mitaka needs, so the version may mismatch at some point.

[Logan V (loganv)]

Option 2 makes sense now. Don't install python-libvirt from repo container wheel, install from OS package that matches the libvirt we are also sourcing from the OS package, and the link the OS packaged libvirt-python module into the venv.

[Logan V (loganv)]

I've got a WIP review for a fix up at https://review.openstack.org/#/c/396821/

If there are no major concerns with the approach, my other question would be regarding CentOS: do we want to continue with the current wheel build/install flow on centos, or should we also move centos to use the distro package using a similar mechanism as this review?

[Logan V (loganv)]

Scratch the WIP comment.. I proposed a solution for CentOS in the review.

[Logan V (loganv)]

To note.. this bug stems from the discussion here: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2016-11-11.log.html#t2016-11-11T14:20:23

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/445995

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/444445
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=fe3ae20f74a912925d5c78040984957a6d55f9de
Submitter: Jenkins
Branch: master

commit fe3ae20f74a912925d5c78040984957a6d55f9de
Author: Kevin Carter <email address hidden>
Date: Fri Mar 10 12:18:08 2017 -0600

    Modified repo build for better multi-OS support

    The repo build role will now store all wheels and venvs in directories
    containg information about the OS in which the bits were build on. The
    intention here is to ensure that all installation are able to support
    multi-OS without running into issues caused by system packages and
    different core libs. This change builds upon the multi-archetecture
    support we already have.

    Closes-Bug: 1641131
    Change-Id: I3f36afa307e02a38d73b860fc23ed94a09882c34
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/445995
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=cacf41028b1fe398175d34cc298ce39042edf09e
Submitter: Jenkins
Branch: master

commit cacf41028b1fe398175d34cc298ce39042edf09e
Author: Kevin Carter <email address hidden>
Date: Wed Mar 15 09:41:51 2017 -0500

    Add better support for multi-OS deployments

    This change adds a few variables which allow it to consume a repo-server
    setup which has releases based on OS and ARCH. The changes require a
    deployer to have at least one repo-server per-OS type being deployed.
    This means that if a deployer wanted to do a mixed CentOS 7 and Ubuntu
    16.04 deployment they'd need at least one repo server in each of OS type
    which facilitates the building of wheels for each OS.

    Change-Id: Ifd5ff22ae47bc9a2a88c1bbb2632ce48ba5accbf
    Closes-Bug: 1641131
    Depends-On: I3f36afa307e02a38d73b860fc23ed94a09882c34
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/446255

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/446256

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/446257

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/446258

[Tom (tomjosekal)]

Will this fix, help in using different kernel.. My env is in stable/newton without this fix.
All my controller nodes are in "4.4.0-67-generic" Kernel with ubuntu 14.04 OS.. And I have two compute hosts with "3.13.0-98-generic" with ubuntu 14.04
While running the setup-host I got the below error:

TASK [openstack_hosts : Check Kernel Version] **********************************
fatal: [node:6]: FAILED! => {"changed": false, "failed": true, "msg": "Wrong kernel Version found [ 3.13.0-98-generic < 4.2.0-18-generic ] Resolve this issue before continuing.\n"}

https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/tasks/openstack_kernel_check.yml#L16-L22

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/446257
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=00fd4897853589beeeef099d70c9aca282bef66f
Submitter: Jenkins
Branch: stable/ocata

commit 00fd4897853589beeeef099d70c9aca282bef66f
Author: Kevin Carter <email address hidden>
Date: Fri Mar 10 12:18:08 2017 -0600

    Modified repo build for better multi-OS support

    The repo build role will now store all wheels and venvs in directories
    containg information about the OS in which the bits were build on. The
    intention here is to ensure that all installation are able to support
    multi-OS without running into issues caused by system packages and
    different core libs. This change builds upon the multi-archetecture
    support we already have.

    Closes-Bug: 1641131
    Change-Id: I3f36afa307e02a38d73b860fc23ed94a09882c34
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit fe3ae20f74a912925d5c78040984957a6d55f9de)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/446256
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-repo_build/commit/?id=3e6345f43768ae0672048072a8dc5bea1f109562
Submitter: Jenkins
Branch: stable/newton

commit 3e6345f43768ae0672048072a8dc5bea1f109562
Author: Kevin Carter <email address hidden>
Date: Fri Mar 10 12:18:08 2017 -0600

    Modified repo build for better multi-OS support

    The repo build role will now store all wheels and venvs in directories
    containg information about the OS in which the bits were build on. The
    intention here is to ensure that all installation are able to support
    multi-OS without running into issues caused by system packages and
    different core libs. This change builds upon the multi-archetecture
    support we already have.

    Closes-Bug: 1641131
    Change-Id: I3f36afa307e02a38d73b860fc23ed94a09882c34
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit fe3ae20f74a912925d5c78040984957a6d55f9de)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/446255
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=510ebc5ad183331b3c4862c0915c9fadcce5987f
Submitter: Jenkins
Branch: stable/ocata

commit 510ebc5ad183331b3c4862c0915c9fadcce5987f
Author: Kevin Carter <email address hidden>
Date: Wed Mar 15 09:41:51 2017 -0500

    Add better support for multi-OS deployments

    This change adds a few variables which allow it to consume a repo-server
    setup which has releases based on OS and ARCH. The changes require a
    deployer to have at least one repo-server per-OS type being deployed.
    This means that if a deployer wanted to do a mixed CentOS 7 and Ubuntu
    16.04 deployment they'd need at least one repo server in each of OS type
    which facilitates the building of wheels for each OS.

    Change-Id: Ifd5ff22ae47bc9a2a88c1bbb2632ce48ba5accbf
    Closes-Bug: 1641131
    Depends-On: I3f36afa307e02a38d73b860fc23ed94a09882c34
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit cacf41028b1fe398175d34cc298ce39042edf09e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/446258
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=78eddadcd31b911158dda0eb50a41a31e8be6ec7
Submitter: Jenkins
Branch: stable/newton

commit 78eddadcd31b911158dda0eb50a41a31e8be6ec7
Author: Kevin Carter <email address hidden>
Date: Wed Mar 15 09:41:51 2017 -0500

    Add better support for multi-OS deployments

    This change adds a few variables which allow it to consume a repo-server
    setup which has releases based on OS and ARCH. The changes require a
    deployer to have at least one repo-server per-OS type being deployed.
    This means that if a deployer wanted to do a mixed CentOS 7 and Ubuntu
    16.04 deployment they'd need at least one repo server in each of OS type
    which facilitates the building of wheels for each OS.

    Change-Id: Ifd5ff22ae47bc9a2a88c1bbb2632ce48ba5accbf
    Closes-Bug: 1641131
    Depends-On: I3f36afa307e02a38d73b860fc23ed94a09882c34
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit cacf41028b1fe398175d34cc298ce39042edf09e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 15.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-repo_build 15.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 14.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-repo_build 14.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 16.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-repo_build 16.0.0.0b1 development milestone.