Update of neutron-server breaks compatibility to previous neutron-agent version

[Christian Rohmann]

During the update of Neutron from 15.1 to 15.3 (Train) server and agent are not compatible (RPC format) anymore.

I was able to narrow down the issue to commit https://opendev.org/openstack/neutron/commit/00298fe6e84cd7610b39af50e9517885a182f47c or patchset https://review.opendev.org/#/c/744133/
With this commit, fixing the security issue in bug https://bugs.launchpad.net/neutron/+bug/1867119 was addressed and a change to the RPC data structure now containing also the MAC address was required consequently back-ported to Train (https://bugs.launchpad.net/neutron/+bug/1867119/comments/6).

This change breaks compatibility as the loop in method "_sanitize_addresses" (neutron/agent/linux/ipset_manager.py", line 47) does not handle this new format properly.

--- 15.1 ---
for ip in addresses:
            ip = netaddr.IPNetwork(ip)
            if ip.prefixlen == 0:
--- /15.1 ---

--- 15.3 ---
for ip, _mac in addresses:
            ip = netaddr.IPNetwork(ip)
            if ip.prefixlen == 0:
--- /15.3 ---

As neutron server in version 15.1 delivers just a list of addresses, i,e. "['192.168.100.57', '192.168.100.160']", while after the update to 15.3 it sends a list of lists with each list element containing an IP as well a MAC address, i.e. "[['192.168.100.57', None], ['192.168.100.160', None]])" the code on the agent side has no way of handling both cases.

According to https://docs.openstack.org/neutron/latest/contributor/internals/upgrade.html .... which states:

"
Those requirements are achieved in Neutron by:

    If the Neutron backend makes use of Neutron agents, the Neutron server have backwards compatibility code to deal with older messaging payloads.

    isolating a single service that accesses database (neutron-server).

To simplify the matter, it’s always assumed that the order of service upgrades is as following:

    first, all neutron-servers are upgraded.

    then, if applicable, neutron agents are upgraded.

This approach allows us to avoid backwards compatibility code on agent side and is in line with other OpenStack projects that support rolling upgrades (specifically, nova).
"

an upgraded neutron-server should still work with the previous neutron-agent.

I took the liberty to flag this as "security vulnerability" as security groups will likely not be applied / managed properly when running mixed between 15.1 and 15.3 which might be a common case in larger clusters.

[Christian Rohmann (christian-rohmann)]

To not cause any confusion, especially regarding the security implications - I suspect that this only becomes an issue when the neutron-agent is running in version 15.3 while the server is still on 15.1:

--- cut ---
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 160, in wrapper
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 207, in process_network_devices
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent device_info.get('updated'))
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.prepare_devices_filter(new_devices)
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 123, in decorated_function
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent *args, **kwargs)
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 135, in prepare_devices_filter
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._apply_port_filter(device_ids)
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 159, in _apply_port_filter
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent security_groups, security_group_member_ips)
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 176, in _update_security_group_info
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent remote_sg_id, member_ips)
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/iptables_firewall.py", line 150, in update_security_group_members
2020-11-03 06:25:10.217 2749 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._update_ipset_members(sg_id, sg_members)
2020-1...

[Christian Rohmann (christian-rohmann)]

Unfortunately I might have been mistaken, it also breaks a 15.1 agent talking to a 15.3 server:

--- cut ---
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-2551d530-3c9f-4041-b1cc-8da49fc13d44 - - - - -] Error in agent loop. Devices info: {'current': {'tap4e9ba928-c4', 'tapd7a77f0d-03', 'tap5299fccd-87', 'tapbfbb2f4e-1d', 'tap29dd4285-61', 'tapb3896a2f-f2'}, 'timestamps': {'tap4e9ba928-c4': 348, 'tapd7a77f0d-03': 353, 'tap5299fccd-87': 351, 'tapbfbb2f4e-1d': 344, 'tap29dd4285-61': 352, 'tapb3896a2f-f2': 347}, 'added': {'tap4e9ba928-c4', 'tapd7a77f0d-03', 'tap5299fccd-87', 'tapbfbb2f4e-1d', 'tap29dd4285-61', 'tapb3896a2f-f2'}, 'removed': set(), 'updated': set()}: TypeError: unexpected type <class 'list'> for addr arg
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 160, in wrapper
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 207, in process_network_devices
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent device_info.get('updated'))
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.prepare_devices_filter(new_devices)
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 123, in decorated_function
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent *args, **kwargs)
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 135, in prepare_devices_filter
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._apply_port_filter(device_ids)
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/agent/securitygroups_rpc.py", line 159, in _apply_port_filter
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_agent security_groups, security_group_member_ips)
2020-11-09 13:44:04.810 667819 ERROR neutron.plugins.ml2.drivers.agent._common_...

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jeremy Stanley (fungi)]

I don't expect there's any reasonable exploit scenario where the Neutron upgrade process is going to be initiated by an attacker, so this strikes me as less of a security vulnerability and more of a bug with security implications: server instances in some environments may temporarily lack security group protections, and an attacker who discovers that's the case could try to take advantage of it, but keeping this fact secret is more likely to increase rather than reduce the chances of it happening. It doesn't seem like the good of our users will be served by discussing this report in private, so I propose we switch to our public workflow.

[Christian Rohmann (christian-rohmann)]

Thanks for the quick response - I have no objections against opening this bug to the public.

[Slawek Kaplonski (slaweq)]

Thx for reporting this bug. It seems to be pretty serious issue but I think that it can be public because, as Jeremy said, I also don't think that there can be any possible exploit using this issue.

[Jeremy Stanley (fungi)]

I've switched this bug to Public and marked it as security affecting. It sounds like a serious defect with security implications, and might warrant a security note if it turns out to be unfixable in supported branches for some reason, but I don't think we would issue an advisory about it (class D in our report taxonomy). https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Christian Rohmann (christian-rohmann)]

We did some post-mortem analysis on this issue and this might be MUCH bigger than initially perceived.

When I reported the issue, I expected the issue to be local with the management of ipsets or similar. But if the versions between neutron-server and neutron-agent are not aligned (we even tested 15.1 vs. 15.3 in both combinations on server / agents) all of the IPtables rules on the neutron-agent machine seem to be empty. With a default of "-A neutron-linuxbri-sg-chain -j ACCEPT" this is rendering virtual machines, especially those with a (public) floating IP unprotected:

1) Initial state: Server on 15.1 + Agent on 15.1:

--- iptables ---
# Generated by iptables-save v1.6.1 on Wed Nov 11 15:47:16 2020
*nat
:PREROUTING ACCEPT [398:359518]
:INPUT ACCEPT [200:346945]
:OUTPUT ACCEPT [295:428261]
:POSTROUTING ACCEPT [410:437514]
COMMIT
# Completed on Wed Nov 11 15:47:16 2020
# Generated by iptables-save v1.6.1 on Wed Nov 11 15:47:16 2020
*mangle
:PREROUTING ACCEPT [46089557:72113781199]
:INPUT ACCEPT [45315616:72074961809]
:FORWARD ACCEPT [3515148:142550180]
:OUTPUT ACCEPT [45341407:87933367449]
:POSTROUTING ACCEPT [48856554:88075916270]
COMMIT
# Completed on Wed Nov 11 15:47:16 2020
# Generated by iptables-save v1.6.1 on Wed Nov 11 15:47:16 2020
*raw
:PREROUTING ACCEPT [38156:364519121]
:OUTPUT ACCEPT [36881:401703191]
:neutron-linuxbri-OUTPUT - [0:0]
:neutron-linuxbri-PREROUTING - [0:0]
-A PREROUTING -j neutron-linuxbri-PREROUTING
-A OUTPUT -j neutron-linuxbri-OUTPUT
-A neutron-linuxbri-PREROUTING -m physdev --physdev-in brqdf7fa34f-b8 -m comment --comment "Set zone for e455ff7-4f" -j CT --zone 4097
-A neutron-linuxbri-PREROUTING -i brqdf7fa34f-b8 -m comment --comment "Set zone for e455ff7-4f" -j CT --zone 4097
-A neutron-linuxbri-PREROUTING -m physdev --physdev-in tap2e455ff7-4f -m comment --comment "Set zone for e455ff7-4f" -j CT --zone 4097
COMMIT
# Completed on Wed Nov 11 15:47:16 2020
# Generated by iptables-save v1.6.1 on Wed Nov 11 15:47:16 2020
*filter
:INPUT ACCEPT [37278:364517585]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [36964:401782749]
:neutron-filter-top - [0:0]
:neutron-linuxbri-FORWARD - [0:0]
:neutron-linuxbri-INPUT - [0:0]
:neutron-linuxbri-OUTPUT - [0:0]
:neutron-linuxbri-i2e455ff7-4 - [0:0]
:neutron-linuxbri-local - [0:0]
:neutron-linuxbri-o2e455ff7-4 - [0:0]
:neutron-linuxbri-s2e455ff7-4 - [0:0]
:neutron-linuxbri-sg-chain - [0:0]
:neutron-linuxbri-sg-fallback - [0:0]
-A INPUT -j neutron-linuxbri-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-linuxbri-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-linuxbri-OUTPUT
-A neutron-filter-top -j neutron-linuxbri-local
-A neutron-linuxbri-FORWARD -m physdev --physdev-out tap2e455ff7-4f --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain
-A neutron-linuxbri-FORWARD -m physdev --physdev-in tap2e455ff7-4f --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-linuxbri-sg-chain
-A neutron-linuxbri-FORWARD -m physdev --physdev-out tapdf6bf070-00 --physdev-is-bridged -m comment --comment "Accept all packets whe...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/763023

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/763024

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/763025

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/763026

[Christian Rohmann (christian-rohmann)]

Reverting the change / commit as proposed will cause the exact same issue again just in the opposite direction, won't it?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/763026
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2f6bf5b4321fabf0a7770a75064aa17118d240c1
Submitter: Zuul
Branch: stable/queens

commit 2f6bf5b4321fabf0a7770a75064aa17118d240c1
Author: Slawek Kaplonski <email address hidden>
Date: Tue Nov 17 14:34:08 2020 +0000

    Revert "[Security] fix allowed-address-pair 0.0.0.0/0 issue"

    This reverts commit bd6203b2c7e1e4af63813b307bc4ec1b49516bd5.

    Change-Id: Ida8a8848813f06dff7a8d98bb0264534dde37573
    Closes-Bug: #1903531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/763025
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b0507fd0e1160493cc5410c044a8cf0eb97be1a9
Submitter: Zuul
Branch: stable/rocky

commit b0507fd0e1160493cc5410c044a8cf0eb97be1a9
Author: Slawek Kaplonski <email address hidden>
Date: Tue Nov 17 14:34:03 2020 +0000

    Revert "[Security] fix allowed-address-pair 0.0.0.0/0 issue"

    This reverts commit 64a0916d2b97700c8f0401ab284684569deebe78.

    Change-Id: I2f476bc66b83232e7d40dad33c230be35897eddd
    Closes-Bug: #1903531

[Slawek Kaplonski (slaweq)]

I spent few hours on it today.
I was trying to reproduce that issue locally with agent on compute node run with code without that patch and server with it.
I have 2 problems with that:
- I wasn't able to reproduce that issue as agent is using CacheBackedPluginApi I have always run method _select_ips_for_remote_group() from SecurityGroupServerAPIShim and that runs locally on agents' side so it has "correct" code and there is no issue like You described.

- as You provided stacktrace of the error, it would be easy to fix it with some try..except TypeError ... block but that would need to be on agent's side so You would need to update agent to consume this fix but if You updated agent already, You don't need that potential fix anymore, right?

So I think that the best idea would be to either:
a) revert this patch to not break deployments which didn't update yet and so if they will update to the newest release they will not break,
b) don't revert anything and just left it as it is with release note about that problem which may potentially happens.

What do You think about it? Do You have maybe any other ideas?

@Christian Rohmann - can You give me more detailed informations about how to reproduce that issue? Maybe then I will be able to figure out some smarter solution.

[Christian Rohmann (christian-rohmann)]

Thanks for digging into this issue further!

I just started my test bed again and can reproduce the issue ...

1) First of all be sure to enable the ipset feature to hold the ip addresses for a security group

2) I added 3 VMs to a network - two of those even having additional floating IPs - but that's likely not important.

3) To hit this code path one needs to add a rule to a security group pointing to a "Remote Security Group" (can even be the same group). This remove security group of course needs to contain interfaces / ip addresses.

[Lajos Katona (lajos-katona)]

@Christian Rohman: could you please check if we revert your deployment is ok? Or is that causing more trouble?

[Christian Rohmann (christian-rohmann)]

@Lajos Katona: There seems to be a fundamental misunderstanding of the issue here.
The initial change caused incompatibility between neutron-server and neutron-agent.
This is independent of the combination in 15.1 or 15.3 are running on either side: they are simply incompatible.

Reverting the change will just create yet another version that will be incompatible...
15.1 vs. 15.3 vs. 15.3-reverted will each be incompatible to their previous / following version.

Certainly if someone did not yet update to 15.3,is still running 15.1, would apply the 15.3-reverted version immediately would have no issues as the introduced incompatibility would have been removed from the code again.

But the issue remains with mixed environments, which is very likely for bigger installations as they cannot simply update every neutron-sever + agent at the same time. If one already went through the update to 15.3 (with the potential to have dysfunctional security groups in the meantime) the new version containing the revered change would create the same scenario yet again - rendering security groups potentially broken until neutron-server + neutron-agents are all running the new version.

In short: I don't think this issue is handled very well actually.

[Slawek Kaplonski (slaweq)]

Based on discussion with release and requirements teams on http://eavesdrop.openstack.org/irclogs/%23openstack-requirements/%23openstack-requirements.2020-11-24.log.html#t2020-11-24T21:36:35 it's the best to revert this patch now.
I know it's very bad for people who already updated their envs but we can't do much more there :/ Sorry for that.

[LIU Yulong (dragon889)]

Looks like a partial upgrading of neutron cluster, neutron-server has a newer version while neutron-agents not. And yes, the backported patch caused such RPC incompatible. But IMO, such partial upgrading is not recommended in a production environment, because neutron-server actually does not do real networking data path work, the agents control the data path flows/iptables/routes and so on. Upgrading neutron-server only looks not so much useful. Secondly, I guess maybe we should announce users every time you try to update/restart/online something, try to do it in a test environment first to see if it works as you expected. Finally the upstream team may have no much time to test all the users' cases, and the upstream code sometimes is buggy. Without any test locally and directly push it to the production environment is not a good idea.

[Christian Rohmann (christian-rohmann)]

@LIU Yulong I accept your opinion, but this is not how things are designed and documented or would even work for larger environments:

If I may quote myself (initial post):

---
According to https://docs.openstack.org/neutron/latest/contributor/internals/upgrade.html .... which states:

"
Those requirements are achieved in Neutron by:

    If the Neutron backend makes use of Neutron agents, the Neutron server have backwards compatibility code to deal with older messaging payloads.

    isolating a single service that accesses database (neutron-server).

To simplify the matter, it’s always assumed that the order of service upgrades is as following:

    first, all neutron-servers are upgraded.

    then, if applicable, neutron agents are upgraded.

This approach allows us to avoid backwards compatibility code on agent side and is in line with other OpenStack projects that support rolling upgrades (specifically, nova).
"

---

[Slawek Kaplonski (slaweq)]

@Liu: I don't agree with Your point tbh. We have documented that it should be compatible and should works but we broke it and it's not working as it should.
More I think about it, more I'm convinced that we should revert that patch from master and all other branches too. If we will not do that, we will always have somewhere point where server from "N" version will not be compatible with agents running "N-1" version - and that shouldn't happen never.
So my proposal is like below:
- revert original patch from every branch,
- propose that patch to master again but with provided compatybility between older agents and newer neutron-server.

That's IMO the only good way to go with this.

Proposed revers:
master - https://review.opendev.org/c/openstack/neutron/+/764189
victoria - https://review.opendev.org/c/openstack/neutron/+/764190
ussuri - https://review.opendev.org/c/openstack/neutron/+/764191

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.4.2 release.

[keerthivasan (keerthivassan86)]

Currently I have a setup with Controller in Stein & compute node in Rocky version. Current neutron version is 13.*.

Getting the below error in neutron-linuxbridge-agent log

2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-5881d194-914d-4a6b-8402-0933f73c8030 - - - - -] Error in agent loop. Devices info: {'current': set(['tap9d1e14b4-c2']), 'timestamps': {'tap9d1e14b4-c2': 27}, 'removed': set([]), 'added': set(['tap9d1e14b4-c2']), 'updated': set([])}: TypeError: unexpected type <type 'list'> for addr arg
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstack/venvs/neutron-18.1.18/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 458, in daemon_loop
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstack/venvs/neutron-18.1.18/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstack/venvs/neutron-18.1.18/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 202, in process_network_devices
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent device_info.get('updated'))
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstack/venvs/neutron-18.1.18/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 256, in setup_port_filters
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.prepare_devices_filter(new_devices)
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstack/venvs/neutron-18.1.18/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in decorated_function
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent *args, **kwargs)
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstack/venvs/neutron-18.1.18/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 131, in prepare_devices_filter
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._apply_port_filter(device_ids)
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstack/venvs/neutron-18.1.18/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 155, in _apply_port_filter
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent security_groups, security_group_member_ips)
2020-12-22 07:51:55.795 13339 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/openstac...

[Slawek Kaplonski (slaweq)]

All reverts and fixes merged to stable branches already

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.