In DVR mode on Rocky py2 packages cause

[David Ames]

Bionic Rocky in DVR mode fails to create a VIF:

2018-11-16 05:18:48.256 1409 ERROR os_vif [req-4d108ae3-9b0f-4e06-963e-24d6c44e28eb d7fd89d2b0134d1ab83fedbbe2e8cd35 7251a61997b84632942f9b895b113f40 - 98efe08df8c946c78f40edd2524ca958 98efe08df8c946c78f40edd2524ca958] Failed to unplug vif VIFBridge(active=False,address=fa:16:3e:58:55:4a,bridge_name='qbr15428f9d-38',has_traffic_filtering=True,id=15428f9d-388f-4aa1-87eb-2b5802198198,network=Network(0db326fc-48b0-471e-afab-dde9cbb71952),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap15428f9d-38'): oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-11-16 05:18:48.256 1409 ERROR os_vif Traceback (most recent call last):
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2354, in _build_resources
2018-11-16 05:18:48.256 1409 ERROR os_vif yield resources
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2118, in _build_and_run_instance
2018-11-16 05:18:48.256 1409 ERROR os_vif block_device_info=block_device_info)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3064, in spawn
2018-11-16 05:18:48.256 1409 ERROR os_vif block_device_info=block_device_info)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3442, in _create_image
2018-11-16 05:18:48.256 1409 ERROR os_vif fallback_from_host)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3533, in _create_and_inject_local_root
2018-11-16 05:18:48.256 1409 ERROR os_vif instance, size, fallback_from_host)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7768, in _try_fetch_image_cache
2018-11-16 05:18:48.256 1409 ERROR os_vif trusted_certs=instance.trusted_certs)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 244, in cache
2018-11-16 05:18:48.256 1409 ERROR os_vif *args, **kwargs)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 602, in create_image
2018-11-16 05:18:48.256 1409 ERROR os_vif nova.privsep.path.utime(base)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2018-11-16 05:18:48.256 1409 ERROR os_vif self.start()
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 217, in start
2018-11-16 05:18:48.256 1409 ERROR os_vif channel = daemon.RootwrapClientChannel(context=self)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 327, in __init__
2018-11-16 05:18:48.256 1409 ERROR os_vif raise FailedToDropPrivileges(msg)
2018-11-16 05:18:48.256 1409 ERROR os_vif oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-11-16 05:18:48.256 1409 ERROR os_vif
2018-11-16 05:18:48.256 1409 ERROR os_vif During handling of the above exception, another exception occurred:
2018-11-16 05:18:48.256 1409 ERROR os_vif
2018-11-16 05:18:48.256 1409 ERROR os_vif Traceback (most recent call last):
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 110, in unplug
2018-11-16 05:18:48.256 1409 ERROR os_vif plugin.unplug(vif, instance_info)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 271, in unplug
2018-11-16 05:18:48.256 1409 ERROR os_vif self._unplug_bridge(vif, instance_info)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 233, in _unplug_bridge
2018-11-16 05:18:48.256 1409 ERROR os_vif linux_net.delete_bridge(vif.bridge_name, v1_name)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2018-11-16 05:18:48.256 1409 ERROR os_vif self.start()
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 217, in start
2018-11-16 05:18:48.256 1409 ERROR os_vif channel = daemon.RootwrapClientChannel(context=self)
2018-11-16 05:18:48.256 1409 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 327, in __init__
2018-11-16 05:18:48.256 1409 ERROR os_vif raise FailedToDropPrivileges(msg)
2018-11-16 05:18:48.256 1409 ERROR os_vif oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-11-16 05:18:48.256 1409 ERROR os_vif

This is due to python2 and python3 packages both being installed:
# dpkg -l |grep oslo
ii python-oslo.cache 1.30.1-0ubuntu1~cloud0 all cache storage for Openstack projects - Python 2.7
ii python-oslo.concurrency 3.27.0-0ubuntu1~cloud0 all concurrency and locks for OpenStack projects - Python 2.x
ii python-oslo.config 1:6.4.0-0ubuntu2~cloud0 all Common code for Openstack Projects (configuration API) - Python 2.x
ii python-oslo.context 1:2.21.0-0ubuntu1~cloud0 all WSGI context helpers for OpenStack - Python 2.x
ii python-oslo.db 4.40.0-0ubuntu1~cloud0 all database connectivity to the different backends and helper utils - Python 2.x
ii python-oslo.i18n 3.21.0-0ubuntu1~cloud0 all Oslo Internationalization Utilities - Python 2.x
ii python-oslo.log 3.39.0-0ubuntu1~cloud0 all OpenStack logging configuration library - Python 2.x
ii python-oslo.messaging 8.1.0-0ubuntu1~cloud0 all oslo messaging library - Python 2.x
ii python-oslo.middleware 3.36.0-0ubuntu1~cloud0 all various WSGI middleware components for OpenStack - Python 2.x
ii python-oslo.policy 1.38.1-0ubuntu1~cloud0 all RBAC policy enforcement library for OpenStack - Python 2.x
ii python-oslo.privsep 1.29.0-0ubuntu1~cloud0 all OpenStack library for privilege separation - Python 2.7
ii python-oslo.reports 1.28.0-0ubuntu3~cloud0 all reports serialized in various data types - Python 2.7
ii python-oslo.rootwrap 5.14.1-0ubuntu1~cloud0 all allows fine filtering of shell commands to run as root - Python 2.x
ii python-oslo.serialization 2.27.0-0ubuntu1~cloud0 all utilities for serialization , especially JSON - Python 2.x
ii python-oslo.service 1.31.3-0ubuntu1~cloud0 all library for running OpenStack services - Python 2.x
ii python-oslo.utils 3.36.4-0ubuntu1~cloud0 all set of utility functions for OpenStack - Python 2.x
ii python-oslo.versionedobjects 1.33.3-0ubuntu1~cloud0 all deals with DB schema versions and code expectations - Python 2.x
ii python3-oslo.cache 1.30.1-0ubuntu1~cloud0 all cache storage for Openstack projects - Python 3.x
ii python3-oslo.concurrency 3.27.0-0ubuntu1~cloud0 all oslo.concurrency library - Python 3.x
ii python3-oslo.config 1:6.4.0-0ubuntu2~cloud0 all Common code for Openstack Projects (configuration API) - Python 3.x
ii python3-oslo.context 1:2.21.0-0ubuntu1~cloud0 all WSGI context helpers for OpenStack - Python 3.x
ii python3-oslo.db 4.40.0-0ubuntu1~cloud0 all database connectivity to the different backends and helper utils - Python 3.x
ii python3-oslo.i18n 3.21.0-0ubuntu1~cloud0 all Oslo Internationalization Utilities - Python 3.x
ii python3-oslo.log 3.39.0-0ubuntu1~cloud0 all OpenStack logging configuration library - Python 3.x
ii python3-oslo.messaging 8.1.0-0ubuntu1~cloud0 all oslo messaging library - Python 3.x
ii python3-oslo.middleware 3.36.0-0ubuntu1~cloud0 all various WSGI middleware components for OpenStack - Python 3.x
ii python3-oslo.policy 1.38.1-0ubuntu1~cloud0 all RBAC policy enforcement library for OpenStack - Python 3.x
ii python3-oslo.privsep 1.29.0-0ubuntu1~cloud0 all OpenStack library for privilege separation - Python 3.x
ii python3-oslo.reports 1.28.0-0ubuntu3~cloud0 all reports serialized in various data types - Python 3.x
ii python3-oslo.rootwrap 5.14.1-0ubuntu1~cloud0 all allows fine filtering of shell commands to run as root - Python 3.x
ii python3-oslo.serialization 2.27.0-0ubuntu1~cloud0 all utilities for serialization , especially JSON - Python 3.x
ii python3-oslo.service 1.31.3-0ubuntu1~cloud0 all library for running OpenStack services - Python 3.x
ii python3-oslo.utils 3.36.4-0ubuntu1~cloud0 all set of utility functions for OpenStack - Python 3.x
ii python3-oslo.versionedobjects 1.33.3-0ubuntu1~cloud0 all deals with DB schema versions and code expectations - Python 3.x
ii python3-oslo.vmware 2.31.0-0ubuntu1~cloud0 all VMware library for OpenStack projects - Python 3.x

The python2 packages are installed because neutron-{l3,dhcp,metadata}-agent depend on either py2 or py3.

Make the package dependencies explicit so that we install the python3 versions.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/618565

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618565
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=b404c18a506c4dcb0967b85cbafb0d238bc3f773
Submitter: Zuul
Branch: master

commit b404c18a506c4dcb0967b85cbafb0d238bc3f773
Author: David Ames <email address hidden>
Date: Fri Nov 16 08:55:26 2018 -0800

    Explicitly install dependencies for DVR

    The DVR package neutron-l3-agent depends on python-neutron-fwaas or
    python3-neutron-fwaas. On Rocky without being explicit it will
    incorrectly install the python2 version which in turn installs many
    python2 dependencies.

    This change explicitly adds python3-neutron-fwaas as a dependency on
    Rocky and updates python-neutron-fwaas as a purge package.

    Change-Id: Idb537df84b044e8ea92527a5f56ab06a37b9ffad
    Closes-Bug: #1803744

[Nell Jerram (neil-jerram)]

David, please could you provide more explanation for your initial diagnosis that "This is due to python2 and python3 packages both being installed"? I am seeing a similar call stack, at least as far as the nova.privsep.path.utime(base) call, and wonder if your diagnosis might apply to my situation too.

[David Ames (thedac)]

Neil,

We made that diagnosis because when both the python2 and python3 sets of oslo packages were installed it failed consistently with the above and when we guaranteed only the python3 oslo packages got installed the problem went away.

It is possible more was going on here.