zun-cni-daemon permission issue on ovs hybrid plug

[hongbin]

User reported the following error on virtual-kubelet/openstack-zun [1]:

2020-07-28 07:13:08.837 23254 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'zun-rootwrap', '/etc/zun/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/zun/zun
.conf', '--privsep_context', 'vif_plug_ovs.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmp_pnf0hwm/privsep.sock']
2020-07-28 07:13:09.623 23254 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
2020-07-28 07:13:09.508 23272 INFO oslo.privsep.daemon [-] privsep daemon starting
2020-07-28 07:13:09.520 23272 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2020-07-28 07:13:09.523 23272 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none
2020-07-28 07:13:09.524 23272 INFO oslo.privsep.daemon [-] privsep daemon running as pid 23272
2020-07-28 07:13:10.154 23254 INFO os_vif [-] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:02:1a:bb,bridge_name='qbr8cd87d70-5c',has_traffic_filtering=True,id=8cd
87d70-5cb7-4397-a5a1-7478a676de91,network=Network(a4322dbc-4ffa-4a38-895d-16a32ee1deb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap8cd8
7d70-5c')
2020-07-28 07:13:10.157 23254 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'zun-rootwrap', '/etc/zun/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/zun/zun
.conf', '--privsep_context', 'zun.common.privileged.cni', '--privsep_sock_path', '/tmp/tmpxuvn729u/privsep.sock']
2020-07-28 07:13:10.938 23254 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
2020-07-28 07:13:10.820 23382 INFO oslo.privsep.daemon [-] privsep daemon starting
2020-07-28 07:13:10.823 23382 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2020-07-28 07:13:10.826 23382 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_NET_ADMIN|CAP_SYS_$DMIN|CAP_SYS_PTRACE/none
2020-07-28 07:13:10.826 23382 INFO oslo.privsep.daemon [-] privsep daemon running as pid 23382
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service [-] Error when processing addNetwork request. CNI Params: {'CNI_COMMAND': 'ADD', 'CNI_CONTAINERID': '995dfcab5c850280ddb01f
c407fefda4cf2116b37f130e9071df61de5eb6b5a4', 'CNI_NETNS': '/var/run/netns/cni-0e5ba6ca-74a9-0709-3973-db740eb0634e', 'CNI_ARGS': 'IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NA
ME=5973904c-be15-4c30-9df0-b0d48e8f98f2;K8S_POD_INFRA_CONTAINER_ID=995dfcab5c850280ddb01fc407fefda4cf2116b37f130e9071df61de5eb6b5a4', 'CNI_IFNAME': 'eth0', 'CNI_PATH': '/opt/cni/bin'
}: pyroute2.netlink.exceptions.NetlinkError: (1, 'Operation not permitted')
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service Traceback (most recent call last):
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/local/lib/python3.6/dist-packages/zun/cni/daemon/service.py", line 66, in add
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service vif = self.plugin.add(params)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/local/lib/python3.6/dist-packages/zun/cni/plugins/zun_cni_registry.py", line 46, in add
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service vifs = self._do_work(params, b_base.connect)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/local/lib/python3.6/dist-packages/zun/cni/plugins/zun_cni_registry.py", line 146, in _do_work
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service container_id=params.CNI_CONTAINERID)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/local/lib/python3.6/dist-packages/zun/cni/binding/base.py", line 132, in connect
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service driver.connect(vif, ifname, netns, container_id)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/local/lib/python3.6/dist-packages/zun/cni/binding/bridge.py", line 88, in connect
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service h_br.add_port(host_ifname)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/ipdb/transactional.py", line 209, in __exit__
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service self.commit()
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 1078, in commit
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service raise error
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 769, in commit
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service run(nl.link, 'update', index=i, master=self['index'])
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 504, in _run
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service raise error
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/ipdb/interfaces.py", line 499, in _run
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service return cmd(*argv, **kwarg)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/iproute/linux.py", line 1332, in link
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service msg_flags=msg_flags)
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 373, in nlm_request
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service return tuple(self._genlm_request(*argv, **kwarg))
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 864, in nlm_request
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service callback=callback):
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 376, in get
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service return tuple(self._genlm_get(*argv, **kwarg))
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service File "/usr/lib/python3/dist-packages/pyroute2/netlink/nlsocket.py", line 701, in get
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service raise msg['header']['error']
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service pyroute2.netlink.exceptions.NetlinkError: (1, 'Operation not permitted')
2020-07-28 07:13:12.330 23254 ERROR zun.cni.daemon.service
2020-07-28 07:13:12.333 23254 INFO werkzeug [-] 127.0.0.1 - - [28/Jul/2020 07:13:12] "POST /addNetwork HTTP/1.1" 500 -

[1] https://github.com/virtual-kubelet/openstack-zun/issues/14

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743664
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=2905c488db46c6f7230dc5c1d04db1c4d6a7de67
Submitter: Zuul
Branch: master

commit 2905c488db46c6f7230dc5c1d04db1c4d6a7de67
Author: Hongbin Lu <email address hidden>
Date: Wed Jul 29 02:34:18 2020 +0000

    Fix ovs hybrid plug in zun cni

    Change-Id: I756f2ce5ed8ad5dd394e9862adcc84fdf4d227de
    Closes-Bug: #1889440