| 2020-10-22 22:34:46 |
Nikolay Vinogradov |
description |
I have a cloud with hardware offload enabled and nova-compute deployed with aa-profile-mode: complain:
$ juju config -m openstack nova-compute-kvm-critical aa-profile-mode
complain
I was able to create HW offloaded VMs just fine using instructions given in [1] right after the cloud was deployed.
However after cloud reboot despite the aa-profile-mode is still set to complain, nova-compute services lost the app armor profile (see https://pastebin.canonical.com/p/zCJyM6YJsz/ for the example) and I get "permission denied" for Nova trying to enumerate VF representors on HW offload VM start attempts:
2020-10-22 21:40:59.480 52842 INFO oslo.privsep.daemon [-] privsep daemon running as pid 52842
2020-10-22 21:40:59.892 29877 ERROR os_vif [req-a0509c9d-8824-42f3-a3bc-bfe7765d756a - - - - -] Failed to plug vif VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): vif_plug_ovs.exception.RepresentorNotFound: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.892 29877 ERROR os_vif Traceback (most recent call last):
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 254, in get_representor_port
2020-10-22 21:40:59.892 29877 ERROR os_vif devices = os.listdir(pf_subsystem_file)
2020-10-22 21:40:59.892 29877 ERROR os_vif PermissionError: [Errno 13] Permission denied: '/sys/class/net/enp59s0f0/subsystem'
2020-10-22 21:40:59.892 29877 ERROR os_vif
2020-10-22 21:40:59.892 29877 ERROR os_vif During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.892 29877 ERROR os_vif
2020-10-22 21:40:59.892 29877 ERROR os_vif Traceback (most recent call last):
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2020-10-22 21:40:59.892 29877 ERROR os_vif plugin.plug(vif, instance_info)
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 299, in plug
2020-10-22 21:40:59.892 29877 ERROR os_vif self._plug_vf(vif, instance_info)
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 265, in _plug_vf
2020-10-22 21:40:59.892 29877 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num)
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 256, in get_representor_port
2020-10-22 21:40:59.892 29877 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2020-10-22 21:40:59.892 29877 ERROR os_vif vif_plug_ovs.exception.RepresentorNotFound: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.892 29877 ERROR os_vif
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service [req-a0509c9d-8824-42f3-a3bc-bfe7765d756a - - - - -] Error starting thread.: nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 254, in get_representor_port
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service devices = os.listdir(pf_subsystem_file)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service PermissionError: [Errno 13] Permission denied: '/sys/class/net/enp59s0f0/subsystem'
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service plugin.plug(vif, instance_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 299, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self._plug_vf(vif, instance_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 265, in _plug_vf
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service representor = linux_net.get_representor_port(pf_ifname, vf_num)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 256, in get_representor_port
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service vif_plug_ovs.exception.RepresentorNotFound: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 709, in _plug_os_vif
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service os_vif.plug(vif, instance_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 82, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service raise os_vif.exception.PlugException(vif=vif, err=err)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service os_vif.exception.PlugException: Failed to plug VIF VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 810, in run_service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service service.start()
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/service.py", line 172, in start
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self.manager.init_host()
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1426, in init_host
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self._init_instance(context, instance)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1128, in _init_instance
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1136, in plug_vifs
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self.vif_driver.plug(instance, vif)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 733, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self._plug_os_vif(instance, vif_obj)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 713, in _plug_os_vif
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service raise exception.InternalError(msg)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
[1] https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-hardware-offload.html |
I have a cloud with hardware offload enabled and nova-compute deployed with aa-profile-mode: complain:
$ juju config -m openstack nova-compute-kvm-critical aa-profile-mode
complain
I was able to create HW offloaded VMs just fine using instructions given in [1] right after the cloud was deployed.
However after cloud reboot despite the aa-profile-mode is still set to complain, nova-compute services lost the app armor profile (see https://pastebin.canonical.com/p/zCJyM6YJsz/ for the example) and I get "permission denied" for Nova trying to enumerate VF representors on HW offloaded VM start attempts:
2020-10-22 21:40:59.480 52842 INFO oslo.privsep.daemon [-] privsep daemon running as pid 52842
2020-10-22 21:40:59.892 29877 ERROR os_vif [req-a0509c9d-8824-42f3-a3bc-bfe7765d756a - - - - -] Failed to plug vif VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): vif_plug_ovs.exception.RepresentorNotFound: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.892 29877 ERROR os_vif Traceback (most recent call last):
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 254, in get_representor_port
2020-10-22 21:40:59.892 29877 ERROR os_vif devices = os.listdir(pf_subsystem_file)
2020-10-22 21:40:59.892 29877 ERROR os_vif PermissionError: [Errno 13] Permission denied: '/sys/class/net/enp59s0f0/subsystem'
2020-10-22 21:40:59.892 29877 ERROR os_vif
2020-10-22 21:40:59.892 29877 ERROR os_vif During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.892 29877 ERROR os_vif
2020-10-22 21:40:59.892 29877 ERROR os_vif Traceback (most recent call last):
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2020-10-22 21:40:59.892 29877 ERROR os_vif plugin.plug(vif, instance_info)
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 299, in plug
2020-10-22 21:40:59.892 29877 ERROR os_vif self._plug_vf(vif, instance_info)
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 265, in _plug_vf
2020-10-22 21:40:59.892 29877 ERROR os_vif representor = linux_net.get_representor_port(pf_ifname, vf_num)
2020-10-22 21:40:59.892 29877 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 256, in get_representor_port
2020-10-22 21:40:59.892 29877 ERROR os_vif raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2020-10-22 21:40:59.892 29877 ERROR os_vif vif_plug_ovs.exception.RepresentorNotFound: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.892 29877 ERROR os_vif
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service [req-a0509c9d-8824-42f3-a3bc-bfe7765d756a - - - - -] Error starting thread.: nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 254, in get_representor_port
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service devices = os.listdir(pf_subsystem_file)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service PermissionError: [Errno 13] Permission denied: '/sys/class/net/enp59s0f0/subsystem'
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service plugin.plug(vif, instance_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 299, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self._plug_vf(vif, instance_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 265, in _plug_vf
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service representor = linux_net.get_representor_port(pf_ifname, vf_num)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 256, in get_representor_port
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service raise exception.RepresentorNotFound(ifname=pf_ifname, vf_num=vf_num)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service vif_plug_ovs.exception.RepresentorNotFound: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 709, in _plug_os_vif
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service os_vif.plug(vif, instance_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 82, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service raise os_vif.exception.PlugException(vif=vif, err=err)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service os_vif.exception.PlugException: Failed to plug VIF VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service Traceback (most recent call last):
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 810, in run_service
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service service.start()
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/service.py", line 172, in start
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self.manager.init_host()
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1426, in init_host
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self._init_instance(context, instance)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1128, in _init_instance
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1136, in plug_vifs
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self.vif_driver.plug(instance, vif)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 733, in plug
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service self._plug_os_vif(instance, vif_obj)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 713, in _plug_os_vif
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service raise exception.InternalError(msg)
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=True,address=fa:16:3e:9c:2d:55,dev_address=0000:3b:10.0,dev_type='ethernet',has_traffic_filtering=True,id=e2ac50e9-d059-4893-81e8-152551bfaf7e,network=Network(ad570c61-f59c-433f-9c51-9245b858440a),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: Failed getting representor port for PF enp59s0f0 with 126
2020-10-22 21:40:59.983 29877 ERROR oslo_service.service
[1] https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-hardware-offload.html |
|
