Services not running that should be: nova-compute after disabling live-migration

[Rony Zeidan]

HI,
this is a 4 compute node install, charm rev 296.

After disabling live-migration, nova-compute status report "Services not running that should be: nova-compute"

nova-compute.log:

ERROR os_vif [req-4b1e88c2-6b72-42a4-9e51-bf11bf97849f - - - - -] Failed to plug vif VIFOpenVSwitch(active=True,address=fa:16:3e:70:5c:86,bridge_name='br-int',has_traffic_filtering=True,id=160e31c3-1032-49b1-88b8-9554dfc03b6b,network=Network(a18a7a4b-87e7-4c2a-b841-6c041abed9a3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap160e31c3-10'): oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2019-04-02 08:38:33.908 334289 ERROR os_vif Traceback (most recent call last):
2019-04-02 08:38:33.908 334289 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2019-04-02 08:38:33.908 334289 ERROR os_vif plugin.plug(vif, instance_info)
2019-04-02 08:38:33.908 334289 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 204, in plug
2019-04-02 08:38:33.908 334289 ERROR os_vif self._get_vif_datapath_type(vif))
2019-04-02 08:38:33.908 334289 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2019-04-02 08:38:33.908 334289 ERROR os_vif self.start()
2019-04-02 08:38:33.908 334289 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 217, in start
2019-04-02 08:38:33.908 334289 ERROR os_vif channel = daemon.RootwrapClientChannel(context=self)
2019-04-02 08:38:33.908 334289 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 327, in __init__
2019-04-02 08:38:33.908 334289 ERROR os_vif raise FailedToDropPrivileges(msg)
2019-04-02 08:38:33.908 334289 ERROR os_vif oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2019-04-02 08:38:33.908 334289 ERROR os_vif
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service [req-4b1e88c2-6b72-42a4-9e51-bf11bf97849f - - - - -] Error starting thread.: nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFOpenVSwitch(active=True,address=fa:16:3e:70:5c:86,bridge_name='br-int',has_traffic_filtering=True,id=160e31c3-1032-49b1-88b8-9554dfc03b6b,network=Network(a18a7a4b-87e7-4c2a-b841-6c041abed9a3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap160e31c3-10'). Got error: privsep helper command exited non-zero (1)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service Traceback (most recent call last):
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service plugin.plug(vif, instance_info)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 204, in plug
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service self._get_vif_datapath_type(vif))
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 206, in _wrap
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service self.start()
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 217, in start
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service channel = daemon.RootwrapClientChannel(context=self)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 327, in __init__
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service raise FailedToDropPrivileges(msg)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service Traceback (most recent call last):
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 783, in _plug_os_vif
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service os_vif.plug(vif, instance_info)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 82, in plug
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service raise os_vif.exception.PlugException(vif=vif, err=err)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service os_vif.exception.PlugException: Failed to plug VIF VIFOpenVSwitch(active=True,address=fa:16:3e:70:5c:86,bridge_name='br-int',has_traffic_filtering=True,id=160e31c3-1032-49b1-88b8-9554dfc03b6b,network=Network(a18a7a4b-87e7-4c2a-b841-6c041abed9a3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap160e31c3-10'). Got error: privsep helper command exited non-zero (1)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service Traceback (most recent call last):
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 794, in run_service
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service service.start()
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/service.py", line 162, in start
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service self.manager.init_host()
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1214, in init_host
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service self._init_instance(context, instance)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 961, in _init_instance
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 819, in plug_vifs
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service self.vif_driver.plug(instance, vif)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 807, in plug
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service self._plug_os_vif(instance, vif_obj)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 787, in _plug_os_vif
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service raise exception.InternalError(msg)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFOpenVSwitch(active=True,address=fa:16:3e:70:5c:86,bridge_name='br-int',has_traffic_filtering=True,id=160e31c3-1032-49b1-88b8-9554dfc03b6b,network=Network(a18a7a4b-87e7-4c2a-b841-6c041abed9a3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap160e31c3-10'). Got error: privsep helper command exited non-zero (1)
2019-04-02 08:38:33.948 334289 ERROR oslo_service.service

And in privsep-helper.log:

2019-04-02 08:33:17.933 319931 CRITICAL privsep [-] Unhandled error: ImportError: No module named vif_plug_ovs.privsep
2019-04-02 08:33:17.933 319931 ERROR privsep Traceback (most recent call last):
2019-04-02 08:33:17.933 319931 ERROR privsep File "/usr/bin/privsep-helper", line 10, in <module>
2019-04-02 08:33:17.933 319931 ERROR privsep sys.exit(helper_main())
2019-04-02 08:33:17.933 319931 ERROR privsep File "/usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 480, in helper_main
2019-04-02 08:33:17.933 319931 ERROR privsep context = importutils.import_class(cfg.CONF.privsep_context)
2019-04-02 08:33:17.933 319931 ERROR privsep File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 30, in import_class
2019-04-02 08:33:17.933 319931 ERROR privsep __import__(mod_str)
2019-04-02 08:33:17.933 319931 ERROR privsep ImportError: No module named vif_plug_ovs.privsep
2019-04-02 08:33:17.933 319931 ERROR privsep

i get the same errors if i try to restart nova-compute service.

[Sahid Orentino (sahid-ferdjaoui)]

Hello Rony,

Thanks for the reported issue.

Can you provide the juju crashdump for further investigation?

[Launchpad Janitor (janitor)]

[Expired for OpenStack nova-compute charm because there has been no activity for 60 days.]

[Celia Wang (ziyiwang)]

I run into this issue too while I'm doing openstack-upgrade from Queens to Rocky.

Steps:
juju config nova-compute-kvm action-managed-upgrade=true openstack-origin=cloud:bionic-rocky
juju run-action --wait nova-compute-kvm/<some unit> openstack-upgrade

After upgrade, nova-compute-kvm unit goes into "blocked" state with msg "Services not running that should be: nova-compute".

nova-compute.log and privsep-helper.log are very similar to what Rony saw.

[Celia Wang (ziyiwang)]

$ dpkg -l |grep vif
ii python3-os-vif 1.9.0-0ubuntu1 all Integration library between network and compute - Python 3.x

Python3-os-vif is installed but in privsep-helper.log, I can see it's looking for python-os-vif.

[Celia Wang (ziyiwang)]

$ sudo update-alternatives --config privsep-helper
There are 2 choices for the alternative privsep-helper (providing /usr/bin/privsep-helper).

  Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/bin/python2-privsep-helper 300 auto mode
  1 /usr/bin/python2-privsep-helper 300 manual mode
  2 /usr/bin/python3-privsep-helper 200 manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/bin/python3-privsep-helper to provide /usr/bin/privsep-helper (privsep-helper) in manual mode

With the alternative manually change from python2 to python3, nova-compute service can start up.

[Celia Wang (ziyiwang)]

Can this be fixed or at least documented in the openstack upgrade doc page?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/821725

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/821725
Committed: https://opendev.org/openstack/charm-nova-compute/commit/5da120ad5d4c37466cff292cfe2ab48729e00d4b
Submitter: "Zuul (22348)"
Branch: master

commit 5da120ad5d4c37466cff292cfe2ab48729e00d4b
Author: Felipe Reyes <email address hidden>
Date: Tue Dec 14 12:47:24 2021 -0300

    Remove python-oslo.privsep on upgrades to >=rocky

    Ubuntu OpenStack Rocky migrated from python2 to python3, the package
    python-oslo.privsep provides the privsep-helper program as an
    alternative, when having it installed next to python3-oslo.privsep the
    first installed will stay as a default.

    This change add python-oslo.privsep to the list of packages that need to
    be purged on OpenStack>=rocky.

    Change-Id: Ib64dfa30b6705a45f82051dfc3634740a5f3a661
    Closes-Bug: #1822763