neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface

[Felipe Reyes]

When deploying OpenStack Zed on Kinetic, the neutron-ovn-metadata-agent fails to provision correctly the datapath for the metadata service.

[Platform]

- s390x
- kinetic
- ovn
- Linux s4lp5 5.19.0-23-generic #24-Ubuntu SMP Fri Oct 14 15:39:36 UTC 2022 s390x s390x s390x GNU/Linux

ii openvswitch-common 3.0.0-0ubuntu1 s390x Open vSwitch common components
ii linux-image-generic 5.19.0.23.22 s390x Generic Linux kernel image

Stacktrace:

[-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface tapf5bae660-31, namespace ovnmeta-f5bae660-31f4-46ce-a93e-279bd493f6bc.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ovsdbapp/event.py", line 177, in notify_loop
    match.run(event, row, updates)
  File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 90, in run
    self.agent.update_datapath(str(row.datapath.uuid), net_name)
  File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 392, in update_datapath
    self.provision_datapath(datapath, net_name)
  File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 487, in provision_datapath
    ip2.addr.delete(ipaddr)
  File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 540, in delete
    delete_ip_address(cidr, self.name, self._parent.namespace)
  File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
    privileged.delete_ip_address(
  File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 271, in _wrap
    return self.channel.remote_call(name, args, kwargs,
  File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 215, in remote_call
    raise exc_type(*result[2])
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface tapf5bae660-31, namespace ovnmeta-f5bae660-31f4-46ce-a93e-279bd493f6bc.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/863779

[Elvira García Ruiz (elviragr)]

Hi, I assigned this to you since you already have a patch proposed.

Kind Regards!

[Frode Nordahl (fnordahl)]

fwiw; I see this on kinetic with a regular deploy on amd64 too, so this is not environment specific. Something has changed that needs to be dealt with in general

[Frode Nordahl (fnordahl)]

Adding this instrumentation prior to the `_run_iproute_addr` call:

    LOG.info(f'HELLO? op "delete" device={device}, namespace={namespace}, ip={ip}, prefixlen={prefixlen}, family={family}')

Reveals:
2022-11-17 07:23:15.619 916107 INFO neutron.privileged.agent.linux.ip_lib [-] HELLO? op "delete" device=tap8b48b83e-51, namespace=ovnmeta-8b48b83e-5925-4d24-9c61-a2633eb377f4, ip=fe80::a85c:30ff:fee3:53e4, prefixlen=64, family=10

Which is weird, because the following command succeeds:
sudo ip netns exec ovnmeta-8b48b83e-5925-4d24-9c61-a2633eb377f4 ip a del fe80::a85c:30ff:fee3:53e4/64 dev tap8b48b83e-51

Does pyroute2 need an update?

[Brian Haley (brian-haley)]

Thanks for the info Frode. I wonder if it's the version of pyroute2? I created a small python program that should run the same commands, perhaps we can give that a try? I can pull a kinetic iso and try if you don't have one running. The below did work on my 22.04-ish system.

# ip netns add foo
# python ./ipd.py

ipd.py:

import socket
import pyroute2
from pyroute2 import netns

def main():
    with pyroute2.NetNS('foo', flags=0) as ip:
        link_id = ip.link_lookup(ifname='lo')
        dev = link_id[0]
        ip.addr("add", index=dev, address='fe80::1', mask=64, family=socket.AF_INET6)
        ip.addr("delete", index=dev, address='fe80::1', mask=64, family=socket.AF_INET6)

if __name__ == "__main__":
    main()

[Brian Haley (brian-haley)]

So I can confirm that on kinetic that simple test fails.

bhaley@virtualbox-22-10:~$ sudo python ./ipd.py
Traceback (most recent call last):
  File "/home/bhaley/./ipd.py", line 15, in <module>
    main()
  File "/home/bhaley/./ipd.py", line 11, in main
    ip.addr("delete", index=dev, address='fe80::1', mask=64, family=socket.AF_INET6)
  File "/usr/lib/python3/dist-packages/pr2modules/iproute/linux.py", line 1625, in addr
    ret = self.nlm_request(
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 401, in nlm_request
    return tuple(self._genlm_request(*argv, **kwarg))
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 906, in nlm_request
    for msg in self.get(
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 404, in get
    return tuple(self._genlm_get(*argv, **kwarg))
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 732, in get
    raise msg['header']['error']
pr2modules.netlink.exceptions.NetlinkError: (95, 'Operation not supported')

bhaley@virtualbox-22-10:~$ dpkg -l | grep pyroute
ii python3-pyroute2 0.6.7-3 all Python3 Netlink library - full package
ii python3-pyroute2.core 0.6.7-3 all Python3 Netlink library - core
ii python3-pyroute2.ethtool 0.6.7-3 all Python3 Netlink library - ethtool
ii python3-pyroute2.ipdb 0.6.7-3 all Python3 Netlink library - ipdb
ii python3-pyroute2.ipset 0.6.7-3 all Python3 Netlink library - ipset
ii python3-pyroute2.ndb 0.6.7-3 all Python3 Netlink library - ndb
ii python3-pyroute2.nftables 0.6.7-3 all Python3 Netlink library - nftables
ii python3-pyroute2.nslink 0.6.7-3 all Python3 Netlink library - nslink
ii python3-pyroute2.protocols 0.6.7-3 all Python3 Netlink library - protocols

I'm running 0.7.3 locally, maybe this version has bugz.

[Brian Haley (brian-haley)]

Found these:

https://bugs.launchpad.net/ubuntu/+source/pyroute2/+bug/1995469

https://github.com/svinota/pyroute2/issues/923

So this is not a neutron issue, kinetic needs to package a version which has that fix, which would be 0.6.10 or later [0]

[0] https://github.com/svinota/pyroute2/blob/0.6.10/pyroute2.core/pr2modules/iproute/linux.py

[Corey Bryant (corey.bryant)]

Brian, thank you very much for narrowing in on this. Let's track the SRU in bug 1995469. I'll make this bug a dup of that one.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/863779
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.