Upgrade from Queens to Rocky fails

[Liam Young]

Upgrade from Queens to Rocky fails with what looks to be a py2 v py3 error. I think this is caused by have a mix of py2 and py3 packages installed.

2018-11-08 13:10:24.741 479 ERROR os_vif [req-61bd8a82-9b29-48aa-a835-47c1c36bdd6d - - - - -] Failed to plug vif VIFBridge(active=True,address=fa:16:3e:ca:cd:1e,bridge_name='qbr49bac72d-e8',has_traffic_filtering=True,id=49bac72d-e8d8-4278-8aa9-9ae45095f24e,network=Network(e5c3
37f9-6651-49eb-b66b-b443eee41060),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap49bac72d-e8'): oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-11-08 13:10:24.741 479 ERROR os_vif Traceback (most recent call last):
2018-11-08 13:10:24.741 479 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2018-11-08 13:10:24.741 479 ERROR os_vif plugin.plug(vif, instance_info)
2018-11-08 13:10:24.741 479 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 209, in plug
2018-11-08 13:10:24.741 479 ERROR os_vif self._plug_bridge(vif, instance_info)
2018-11-08 13:10:24.741 479 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 161, in _plug_bridge
2018-11-08 13:10:24.741 479 ERROR os_vif linux_net.ensure_bridge(vif.bridge_name)
2018-11-08 13:10:24.741 479 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2018-11-08 13:10:24.741 479 ERROR os_vif self.start()
2018-11-08 13:10:24.741 479 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 217, in start
2018-11-08 13:10:24.741 479 ERROR os_vif channel = daemon.RootwrapClientChannel(context=self)
2018-11-08 13:10:24.741 479 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 327, in __init__
2018-11-08 13:10:24.741 479 ERROR os_vif raise FailedToDropPrivileges(msg)
2018-11-08 13:10:24.741 479 ERROR os_vif oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-11-08 13:10:24.741 479 ERROR os_vif
2018-11-08 13:10:24.749 479 ERROR oslo_service.service [req-61bd8a82-9b29-48aa-a835-47c1c36bdd6d - - - - -] Error starting thread.: nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=True,address=fa:16:3e:ca:cd:1e,bridg
e_name='qbr49bac72d-e8',has_traffic_filtering=True,id=49bac72d-e8d8-4278-8aa9-9ae45095f24e,network=Network(e5c337f9-6651-49eb-b66b-b443eee41060),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap49bac72d-e8'). Got error: privsep helper c
ommand exited non-zero (1)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service Traceback (most recent call last):
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2018-11-08 13:10:24.749 479 ERROR oslo_service.service plugin.plug(vif, instance_info)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 209, in plug
2018-11-08 13:10:24.749 479 ERROR oslo_service.service self._plug_bridge(vif, instance_info)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 161, in _plug_bridge
2018-11-08 13:10:24.749 479 ERROR oslo_service.service linux_net.ensure_bridge(vif.bridge_name)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2018-11-08 13:10:24.749 479 ERROR oslo_service.service self.start()
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 217, in start
2018-11-08 13:10:24.749 479 ERROR oslo_service.service channel = daemon.RootwrapClientChannel(context=self)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 327, in __init__
2018-11-08 13:10:24.749 479 ERROR oslo_service.service raise FailedToDropPrivileges(msg)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service
2018-11-08 13:10:24.749 479 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2018-11-08 13:10:24.749 479 ERROR oslo_service.service
2018-11-08 13:10:24.749 479 ERROR oslo_service.service Traceback (most recent call last):
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 783, in _plug_os_vif
2018-11-08 13:10:24.749 479 ERROR oslo_service.service os_vif.plug(vif, instance_info)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 82, in plug
2018-11-08 13:10:24.749 479 ERROR oslo_service.service raise os_vif.exception.PlugException(vif=vif, err=err)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service os_vif.exception.PlugException: Failed to plug VIF VIFBridge(active=True,address=fa:16:3e:ca:cd:1e,bridge_name='qbr49bac72d-e8',has_traffic_filtering=True,id=49bac72d-e8d8-4278-8aa9-9ae45095f24e,network=Network(e5c337f9-66
51-49eb-b66b-b443eee41060),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap49bac72d-e8'). Got error: privsep helper command exited non-zero (1)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service
2018-11-08 13:10:24.749 479 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2018-11-08 13:10:24.749 479 ERROR oslo_service.service
2018-11-08 13:10:24.749 479 ERROR oslo_service.service Traceback (most recent call last):
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 794, in run_service
2018-11-08 13:10:24.749 479 ERROR oslo_service.service service.start()
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/service.py", line 162, in start
2018-11-08 13:10:24.749 479 ERROR oslo_service.service self.manager.init_host()
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1211, in init_host
2018-11-08 13:10:24.749 479 ERROR oslo_service.service self._init_instance(context, instance)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 958, in _init_instance
2018-11-08 13:10:24.749 479 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 819, in plug_vifs
2018-11-08 13:10:24.749 479 ERROR oslo_service.service self.vif_driver.plug(instance, vif)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 807, in plug
2018-11-08 13:10:24.749 479 ERROR oslo_service.service self._plug_os_vif(instance, vif_obj)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 787, in _plug_os_vif
2018-11-08 13:10:24.749 479 ERROR oslo_service.service raise exception.InternalError(msg)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=True,address=fa:16:3e:ca:cd:1e,bridge_name='qbr49bac72d-e8',has_traffic_filtering=True,id=49bac72d-e8d8-4278-8aa9
-9ae45095f24e,network=Network(e5c337f9-6651-49eb-b66b-b443eee41060),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap49bac72d-e8'). Got error: privsep helper command exited non-zero (1)
2018-11-08 13:10:24.749 479 ERROR oslo_service.service

[Liam Young (gnuoy)]

# dpkg -l | grep -E 'privsep|rootwrap'
ii python-oslo.rootwrap 5.14.1-0ubuntu1~cloud0 all allows fine filtering of shell commands to run as root - Python 2.x
ii python3-oslo.privsep 1.29.0-0ubuntu1~cloud0 all OpenStack library for privilege separation - Python 3.x
ii python3-oslo.rootwrap 5.14.1-0ubuntu1~cloud0 all allows fine filtering of shell commands to run as root - Python 3.x

[Liam Young (gnuoy)]

Start-Date: 2018-11-08 13:05:59
Commandline: apt-get --assume-yes --option=Dpkg::Options::=--force-confold install ubuntu-cloud-keyring
--
Start-Date: 2018-11-08 13:06:21
Commandline: apt-get --assume-yes --option Dpkg::Options::=--force-confnew --option Dpkg::Options::=--force-confdef dist-upgrade
--
Start-Date: 2018-11-08 13:08:16
Commandline: apt-get --assume-yes --option=Dpkg::Options::=--force-confold install nova-compute genisoimage librbd1 xfsprogs ceph-common nova-compute-kvm python3-nova python3-memcache
--
Start-Date: 2018-11-08 13:09:24
Commandline: apt-get --assume-yes purge python-nova
--
Start-Date: 2018-11-08 13:09:27
Commandline: apt-get --assume-yes autoremove --purge
--
Start-Date: 2018-11-08 13:10:44
Commandline: apt-get --assume-yes --option=Dpkg::Options::=--force-confold install python3-ceilometer
--
Start-Date: 2018-11-08 13:11:27
Commandline: apt-get --assume-yes --option=Dpkg::Options::=--force-confold install python3-neutron
--
Start-Date: 2018-11-08 13:11:42
Commandline: apt-get --assume-yes purge python-neutron
--
Start-Date: 2018-11-08 13:11:46
Commandline: apt-get --assume-yes autoremove --purge

[Liam Young (gnuoy)]

It looks like nova-compute upgrades nova and sends a remote restart trigger to neutron-openvswitch. n-ovs reacts and realises it needs to upgrade its packages. While neutron is upgrading nova charm restarts nova-compute and boom

[Alvaro Uria (aluria)]

Fixed by running:
apt-get remove --purge python-oslo.privsep

[Alvaro Uria (aluria)]

BTW, I hit it when using nova-compute 19.04, on an environment that started as icehouse back in the days, and it was lately upgraded to Queens, then Rocky. When in Rocky, comment #4 helped fix the problem (nova-compute service died because of the error described in the bug).

[Drew Freiberger (afreiberger)]

When I encountered this just now with 21.04 charms, it appears that the real resolution is that update-alternatives switches privsep-helper from py2 to py3 when you purge the python-oslo.privsep package.

Suggest that the python3-oslo.privsep package in UCA:bionic-rocky should force update-alternatives.

I see that python-oslo.privsep is prevented from being removed by the charm because python-os-brick relies on it.

[Przemyslaw Lal (przemeklal)]

I ran into the same issue while upgrading from bionic-queens to bionic-rocky.

As a workaround I ran update-alternatives to switch to the python3 version then purged python-oslo.privsep.

[Takashi Kajinami (kajinamit)]

So this looks like an issue with charm, not one with oslo.privsep.