Linuxbridge agent NetlinkError: (13, 'Permission denied') after Stein upgrade

[Adrien Cunin]

OpenStack-Ansible deployment, just upgraded to Stein, using Ubuntu 18.04.

Neutron Linuxbridge agent logs keep complaining about NetlinkError: (13, 'Permission denied').
After enabling debug, we actually see two stack traces, one happening in privsep, see log attached.

We found out disabling IPv6 on the host workarounds the problem, but that's not a proper fix of course.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Adrien:

Can you add debug logs in the "add_ip_address" method [1]? That will help to debug this problem. Once we have the interface and the parameters (IP address, scope, etc) that trigger this error, you can try to do this manually in the host ("ip addr add dev ...")

You said that disabling IPv6 workarounds the problem. Do you have IPv6 enabled in the kernel [2]? Do you have selinux properly configured?

Regards.

[1]https://github.com/openstack/neutron/blob/b63d1d5a5125f1110fb5b257aa9f1f14d0787864/neutron/privileged/agent/linux/ip_lib.py#L302-L317
[2]https://access.redhat.com/solutions/3559431

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

There is a clash between [1] and [2]. The second one is calling the first one, "get_devices_with_ip", but in this method we are forcing again the conversion of the scope from a string ("link", "global", etc.) to a pyroute2 value (see "rtnl.rtscopes").

IMO, the best solution is undoing the conversion done in [2]. This is the other only call to "get_devices_with_ip" with "scope" parameter. I always prefer to make those parameter conversions close to the library method.

Regards.

[1]https://review.opendev.org/#/c/747628/1/neutron/agent/linux/ip_lib.py
[2]https://github.com/openstack/neutron/blob/b3582e1ae3627bc0a8b35238dddc90647a5f9d89/neutron/agent/linux/ip_lib.py#L538

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/757107

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

The patch for this bug should be backported up to Stein, due to [1].

[1]https://review.opendev.org/#/q/Ia41c1bc627ad2ce89d658ff1fdedee802f6dfa15

[Igor Gnatenko (i-gnatenko-brain)]

Scratch what I"ve said, your patch works just fine for me. Sorry.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757107
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0705699e6072f0862cee7749de7f7633ecc9dcfd
Submitter: Zuul
Branch: master

commit 0705699e6072f0862cee7749de7f7633ecc9dcfd
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 9 13:26:33 2020 +0000

    "scope" conversion only just before pyroute2 method call

    In "IpAddrCommand.list" method, the "scope" parameter is a string
    ("link", "site", "global" or "host"). This method will retrieve all
    devices with an IP address calling "ip_lib.get_devices_with_ip".

    Since [1], "ip_lib.get_devices_with_ip" makes the conversion of
    "scope" string parameter to pyroute2 format (see
    "pyroute2.netlink.rtnl.rtscopes"). The list command should skip then
    the previous conversion.

    Closes-Bug: #1899141

    [1]https://review.opendev.org/#/c/747406/

    Change-Id: I55a0f4341b328af52ea3bd758a72f633fbe3abcb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/759265

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/759266

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/759267

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/759269

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759266
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cd0526dce8703bce12763c5c9bfc51f6d27f16c2
Submitter: Zuul
Branch: stable/ussuri

commit cd0526dce8703bce12763c5c9bfc51f6d27f16c2
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 9 13:26:33 2020 +0000

    "scope" conversion only just before pyroute2 method call

    In "IpAddrCommand.list" method, the "scope" parameter is a string
    ("link", "site", "global" or "host"). This method will retrieve all
    devices with an IP address calling "ip_lib.get_devices_with_ip".

    Since [1], "ip_lib.get_devices_with_ip" makes the conversion of
    "scope" string parameter to pyroute2 format (see
    "pyroute2.netlink.rtnl.rtscopes"). The list command should skip then
    the previous conversion.

    Closes-Bug: #1899141

    [1]https://review.opendev.org/#/c/747406/

    Change-Id: I55a0f4341b328af52ea3bd758a72f633fbe3abcb
    (cherry picked from commit 0705699e6072f0862cee7749de7f7633ecc9dcfd)
    (cherry picked from commit f0849bb0d2a9cbd19be7ee5312ba7e5edbc18900)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759269
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=289952a13f6ecf428eeabaf41020ba9ca459218e
Submitter: Zuul
Branch: stable/stein

commit 289952a13f6ecf428eeabaf41020ba9ca459218e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 9 13:26:33 2020 +0000

    "scope" conversion only just before pyroute2 method call

    In "IpAddrCommand.list" method, the "scope" parameter is a string
    ("link", "site", "global" or "host"). This method will retrieve all
    devices with an IP address calling "ip_lib.get_devices_with_ip".

    Since [1], "ip_lib.get_devices_with_ip" makes the conversion of
    "scope" string parameter to pyroute2 format (see
    "pyroute2.netlink.rtnl.rtscopes"). The list command should skip then
    the previous conversion.

    Closes-Bug: #1899141

    [1]https://review.opendev.org/#/c/747406/

    Change-Id: I55a0f4341b328af52ea3bd758a72f633fbe3abcb
    (cherry picked from commit 0705699e6072f0862cee7749de7f7633ecc9dcfd)
    (cherry picked from commit f0849bb0d2a9cbd19be7ee5312ba7e5edbc18900)
    (cherry picked from commit cd0526dce8703bce12763c5c9bfc51f6d27f16c2)
    (cherry picked from commit 4ef6566b6dc3d069fccf6a7b8d7a8abbadd12d35)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759265
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f0849bb0d2a9cbd19be7ee5312ba7e5edbc18900
Submitter: Zuul
Branch: stable/victoria

commit f0849bb0d2a9cbd19be7ee5312ba7e5edbc18900
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 9 13:26:33 2020 +0000

    "scope" conversion only just before pyroute2 method call

    In "IpAddrCommand.list" method, the "scope" parameter is a string
    ("link", "site", "global" or "host"). This method will retrieve all
    devices with an IP address calling "ip_lib.get_devices_with_ip".

    Since [1], "ip_lib.get_devices_with_ip" makes the conversion of
    "scope" string parameter to pyroute2 format (see
    "pyroute2.netlink.rtnl.rtscopes"). The list command should skip then
    the previous conversion.

    Closes-Bug: #1899141

    [1]https://review.opendev.org/#/c/747406/

    Change-Id: I55a0f4341b328af52ea3bd758a72f633fbe3abcb
    (cherry picked from commit 0705699e6072f0862cee7749de7f7633ecc9dcfd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759267
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4ef6566b6dc3d069fccf6a7b8d7a8abbadd12d35
Submitter: Zuul
Branch: stable/train

commit 4ef6566b6dc3d069fccf6a7b8d7a8abbadd12d35
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 9 13:26:33 2020 +0000

    "scope" conversion only just before pyroute2 method call

    In "IpAddrCommand.list" method, the "scope" parameter is a string
    ("link", "site", "global" or "host"). This method will retrieve all
    devices with an IP address calling "ip_lib.get_devices_with_ip".

    Since [1], "ip_lib.get_devices_with_ip" makes the conversion of
    "scope" string parameter to pyroute2 format (see
    "pyroute2.netlink.rtnl.rtscopes"). The list command should skip then
    the previous conversion.

    Closes-Bug: #1899141

    [1]https://review.opendev.org/#/c/747406/

    Change-Id: I55a0f4341b328af52ea3bd758a72f633fbe3abcb
    (cherry picked from commit 0705699e6072f0862cee7749de7f7633ecc9dcfd)
    (cherry picked from commit f0849bb0d2a9cbd19be7ee5312ba7e5edbc18900)
    (cherry picked from commit cd0526dce8703bce12763c5c9bfc51f6d27f16c2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.