[OSSA-2021-001] Anti-spoofing bypass for Open vSwitch networks (CVE-2021-20267)

[David Sinquin]

Using Open vSwitch on an ussuri setup with neutron 16.0.0, VMs can send ICMPv6 Neighbor Advertisement packets with no check on their content to mis-direct traffic to them.

This looks a lot like https://bugs.launchpad.net/neutron/+bug/1502933 except it affects Open vSwitch driver rather than iptables.

Pre-condition:

- two running VMs in the same L2 flat network with IPv6 connectivity

How to reproduce:

- manually add a custom IPv6 on one (e.g. `ip -6 address add fe80::42/64 dev eth0`)
- ping it from the other, expecting no answer (e.g. `ping -c1 -w1 "fe80::42%eth0"`)
- confirm it updated its neighbor table (e.g. `ip -6 neigh get fe80::42/64 dev eth0`)

Expected behavior:

- VMs should not be able to advertise IPv6 addresses that are not assigned to them e.g. through neighbor advertisement packets.

Affected versions:

The Openstack version I am using is Ussuri with neutron 16.0.0, with minor changes on commit df5b28c2e5. From a quick review of the diff with master, I think the issue is also present there. Network part is using Open vSwitch on flat network with Xen 4.13 as hypervisor.

Similarly, UDP packets using DHCP query ports (for DHCP v4 or v6) can be sent with arbitrary IP and MAC addresses.
And I think we are fine for other ICMP types (redirect, router renumbering for ICMPv6) as I did not managed to have such packets sent between VMs but I fail to understand how it gets filtered.

I am attaching a couple patches that I think fix the issues but include no tests and include changes that we may want to avoid (in case plugins out of neutron git repo use firewall.ICMPV6_ALLOWED_EGRESS_TYPES).

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jeremy Stanley (fungi)]

I think this is an already known issue, judging from abandoned attempts to correct it years ago:

https://review.opendev.org/315828

It got a mention in bug 1502933 which you referenced, as well as other bug reports linked from the commit message there. As such, I think we should be able to safely switch to discussing this report in public. Do Neutron security reviewers agree?

[Slawek Kaplonski (slaweq)]

@fungi: yes, I think this is similar bug to 1502933 and we can make it public IMO.
@David: is this happening for You for openvswitch fw driver or iptables_hybrid? Or both?

[David Sinquin (david-5-1)]

@slaweq: I face the issue using openvswitch fw driver but I did not test iptables_hybrid so I don't know if the issue exists there.

If you all think it should be public, I am OK with that.

[Jeremy Stanley (fungi)]

This report is now being tracked in public, so it can be discussed in public meetings and on mailing lists, and changes related to it can be pushed to normal code review (ideally referencing this bug number).

Pending further confirmation, this sounds likely to be a class A report per our taxonomy (and so would be covered in a security advisory similar to OSSA-2016-009/CVE-2015-8914): https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Marking this patch as "high", same as https://bugs.launchpad.net/neutron/+bug/1502933.

[Slawek Kaplonski (slaweq)]

I was able to reproduce that bug on master. For now I did it with ovs firewall driver.
I will need to confirm if that is also the case for iptables_hybrid fw driver.

[Slawek Kaplonski (slaweq)]

I just checked and I can't reproduce the same issue with iptables_hybrid fw driver.
So it seems that only openvswitch firewall driver is impacted by that (at least on master).

[Slawek Kaplonski (slaweq)]

Patch proposed https://review.opendev.org/c/openstack/neutron/+/776599

[Slawek Kaplonski (slaweq)]

I also tested OVN backend and it seems that it isn't affected. So fix is needed only for ovs firewall driver.

[Slawek Kaplonski (slaweq)]

Patch https://review.opendev.org/c/openstack/neutron/+/776599 merged and it should solves that problem.

[Summer Long (slong-g)]

Fantastic, Slawek, thanks! Jeremy, have you already asked Mitre for a CVE? Otherwise, RH can issue one.

[Jeremy Stanley (fungi)]

Summer: I was hoping to first see some core reviews on the proposed backports before assuming it's going to be fixed there, and as confirmation that we know for sure the extent of the problem prior to the current state of the master branch. The backport targeting stable/victoria looks like it may also fail the neutron-grenade-dvr-multinode upgrade job, it was rechecked a few minutes ago to hopefully rule that out.

Once there's at least one +2 on some of the backports I'll be more confident in drafting an accurate impact description with which to request a CVE assignment. Since this is already public, we're not under any tight embargo timeline with it, and probably not much point in having a CVE to track it until we have a fix we know we can recommend. That said, if you want to act as a CNA and issue a placeholder CVE now, just make sure to let us know the number and I'll refrain from requesting one from MITRE later.

[Summer Long (slong-g)]

Thanks Jeremy, CVE-2021-20267 has been assigned to this issue.

[Jeremy Stanley (fungi)]

The stable/victoria backport has merged and the other backports all have positive votes from from core reviewers. Here's an initial draft for an impact description we'll be able to use in an advisory; please review and call out anything which needs correcting (and David, let me know if you have an employer or other affiliated organization you wish mentioned alongside your name)...

Title: Anti-spoofing bypass for Open vSwitch networks
Reporter: David Sinquin
Products: Neutron
Affects: <15.3.3, >=16.0.0 <16.3.1, >=17.0.0 <17.1.1

Description: David Sinquin reported a vulnerability in Neutron's default Open vSwitch firewall rules. By sending carefully crafted packets, anyone in control of a server instance connected to the virtual switch can impersonate the IPv6 addresses of other systems on the network, resulting in denial of service or in some cases possibly interception of traffic intended for other destinations. Only deployments using the Open vSwitch driver are affected.

[David Sinquin (david-5-1)]

I want to first thank everyone involved.

Regarding credits, please mention “Gandi.net” as my employee (for which I am reporting this and auditing spoofing possibilities in the first place).

Regarding Slawek Kaplonski's patch [1] that was merged in master, I am sorry I did not took time to review this further before, but I **think** there is still a potential security concern (DoS / redirection) for Neighbor Advertisements. I have not tested this using neutron so I cannot guaranty this be possible in a real deployment.

The scenario I think of would require 2 VMs : one “attacker”, one “victim” whose neighbor cache is to be poisoned (for a spoofed IP). The following is an attempt at providing a way to check for the vulnerability:

First, on attacker, add the IP to spoof. It must be unattributed to avoid races with legit owner and simplify testing.

Second, on the attacker, install scapy and start it (use `scapy` to have useful imports) and adapt then run the below lines, leaving the prompt opened for fourth step:

    spoofed_ipv6 = '' # FIXME non-ll (NOT fe80::), L2-reachable
    victim_ipv6 = '' # FIXME must be L2-reachable
    announced_mac = '' # FIXME attacker MAC

    ipv6 = IPv6(dst=victim_ipv6)
    icmpv6na = ICMPv6ND_NA(tgt=spoofed_ipv6)
    icmpv6_target_ll_addr = ICMPv6NDOptDstLLAddr(lladdr=announced_mac)
    spoofed_packet = Ether() / ip6 / icmp6na / icmpv6_target_ll_addr
    spoofed_packet.show() # not mandatory :-)

Third, on the victim, run a tcpdump (e.g. `tcpdump -nvve 'icmp6 && ip6[40] == 134'`) and while it runs, run a ping6 to the unattributed IPv6 to spoof (clear cache before if needed) and immediately continue to the next step.

Fourth, using the scapy prompt on the attacker, run (a few times by security, one should be enough but packet loss might occur):
    sendp(spoofed_packet)

Finally, check the victim tcpdump (it should be empty) and cache with e.g. `ip -6 n` (idem).

(I realize after having finished proofreading this that the same could likely be achieved through some kind of targeted IPv6 NAT for outgoing ICMPv6 packets on the attacker to avoid scapy completely.)

In the patch I had attached to this bug, I was using nd_target to rule this possible issue out but it can only be applied to Neighbor Advertisement packets (since other types do not have such value), hence the other changes I had done around NA management.

[1] https://review.opendev.org/c/openstack/neutron/+/776599

[Gage Hugo (gagehugo)]

I am good with Jeremy's impact description in comment #16.

[David Sinquin (david-5-1)]

Jeremy's impact description in comment #16 also looks good to me FWIW :-)

[Jeremy Stanley (fungi)]

Given David's suggestion (in comment #17) that the merged fix may be incomplete, I'd rather hold off finalizing any advisory until someone can either confirm or refute it.

[Slawek Kaplonski (slaweq)]

Hi David,

Here is what I did exactly to try what You described in the c17: http://paste.openstack.org/show/803502/
For me it seems that I wasn't able to reproduce what You described but maybe I did something wrong there. If so, please provide me reproducer so I can check that.

[David Sinquin (david-5-1)]

Hi Slaweq,

from what I understand, one of the issue you had is the fact your VM had no route to 3000::/64 and therefore no ICMPv6 packets were sent at all.

I also realize the tcpdump command I provided was wrong, and can be replaced with e.g.:
    tcpdump -nve 'icmp6 && (ip6[40] == 135 || ip6[40] == 136)'
(135 is Neighbor Solicitation, 136 is Neighbor Advertisement, 134 was unrelated Router Advertisement…)

I have replied to your paste with what I get (with a reworked procedure I tested in a OpenStack environment), you can find it at: http://paste.openstack.org/show/803529/

The main changes I did are:
 - switching to using fe80::/64 IPs (due to my local setup, I preferred using them)
 - sending spoofed packets in a loop
 - changing NA options to match legitimate answer my VM was sending (likely useless, but it works as-is :-D)
 - some clean-up of unnecessary steps.

[David Sinquin (david-5-1)]

FTR, I did my tests using ussuri neutron version with 4b5bcff64c4725b37f094dfd2613ec58f723d304 cherry-picked on top of 4edf21a37561971991ee450b07fcfbca95922bc2 (plus internal patches that do not change openvswitch driver code).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.1 release.

[Jeremy Stanley (fungi)]

I've asked in #openstack-neutron whether we can switch this back to In Progress state or similar, since it sounds like the fixes were incomplete.

[Slawek Kaplonski (slaweq)]

Thx David. I was able to reproduce that issue with Your script.

[David Sinquin (david-5-1)]

Thx for confirming that :-)

[Slawek Kaplonski (slaweq)]

After some more research I'm not sure if we have any way to fix that issue really as it seems for me that OVS don't allows us to match on tgt field of the ICMPv6 packets and that's where this spoofed IP address is send really in Your example.

If You have any idea on how to fix that, feel free to propose patch for it.

[David Sinquin (david-5-1)]

I think the patch I had attached to this bug report had a working fix for that using nd_target param in _add_flow method.

The code is mostly:

self._add_flow(
    table=ovs_consts.BASE_EGRESS_TABLE,
    priority=95,
    in_port=port.ofport,
    reg_port=port.ofport,
    dl_type=lib_const.ETHERTYPE_IPV6,
    nw_proto=lib_const.PROTO_NUM_IPV6_ICMP,
    icmp_type=n_const.ICMPV6_TYPE_NA,
    nd_target=allowed_ip_addr,
    actions='resubmit(,%d)' % (
        ovs_consts.ACCEPTED_EGRESS_TRAFFIC_NORMAL_TABLE)
)

and results in the following Open vSwitch rule:

table=71, hard_age=65534, priority=95,icmp6,reg5=0x4,in_port=4,icmp_type=136,nd_target=fe80::f816:3eff:fed3:1eb4,actions=resubmit(,94)

The above is as-is allowing the VM to use an IPv6 it does not own to announce its own IPs but can likely be restricted further and does not prevent it from announcing a MAC address it does not own but this requires privileges inside the VM and these privileges are anyway enough to make many other DoS for the VM itself.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[Slawek Kaplonski (slaweq)]

Hi David,

I just proposed Your patch in gerrit https://review.opendev.org/c/openstack/neutron/+/783743 - please take a look and check if I didn't made any mistake there.
I also added some UT for it.

And I tested Your patch locally - it worked for me and I wasn't able to spoof address anymore with that patch using Your script described in comment #22

[David Sinquin (david-5-1)]

Thanks Slaweq, your commit looks good to me and as you tested your version (and I had tested the patch you adapted), I think we are fine.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/783743
Committed: https://opendev.org/openstack/neutron/commit/ca7822e2108c151bda992ef8a6d454ec2c6d890e
Submitter: "Zuul (22348)"
Branch: master

commit ca7822e2108c151bda992ef8a6d454ec2c6d890e
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/791464

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/791465

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/791467

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/791468

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/791500

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/791469

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/791470

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791468
Committed: https://opendev.org/openstack/neutron/commit/2231a9d40f989a1e8a20f9862ff03916091b3215
Submitter: "Zuul (22348)"
Branch: stable/train

commit 2231a9d40f989a1e8a20f9862ff03916091b3215
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791467
Committed: https://opendev.org/openstack/neutron/commit/ac474307d3a800c30ecbabf0f47de931ad310339
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ac474307d3a800c30ecbabf0f47de931ad310339
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791469
Committed: https://opendev.org/openstack/neutron/commit/9f4a9f8f86720567a8ed5e7c5c4430c115ad3c11
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 9f4a9f8f86720567a8ed5e7c5c4430c115ad3c11
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791470
Committed: https://opendev.org/openstack/neutron/commit/ccea0021bc49eefde9af6568942159b52588daae
Submitter: "Zuul (22348)"
Branch: stable/queens

commit ccea0021bc49eefde9af6568942159b52588daae
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791500
Committed: https://opendev.org/openstack/neutron/commit/e1fe735e843b5d1bee4fa18e6011121027422203
Submitter: "Zuul (22348)"
Branch: stable/stein

commit e1fe735e843b5d1bee4fa18e6011121027422203
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791465
Committed: https://opendev.org/openstack/neutron/commit/eb2851561a89338c5843521128cbef3c1c7de10c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit eb2851561a89338c5843521128cbef3c1c7de10c
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791464
Committed: https://opendev.org/openstack/neutron/commit/cb746e2ca454885ba5cf653493b9bb0e91480759
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit cb746e2ca454885ba5cf653493b9bb0e91480759
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ossa/+/800107

[Jeremy Stanley (fungi)]

I'd like to go ahead and publish the advisory for this early next week, if anyone has time to review the proposed content here: https://review.opendev.org/800107

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.1.0 release.

[David Sinquin (david-5-1)]

Thank you fungi, the OSSA looks correct and relevant to me.

And also thanks to everyone involved for the rest of the work :-)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ossa/+/800107
Committed: https://opendev.org/openstack/ossa/commit/239ec3826a74c4f3ffb1239cc574f95c6097c631
Submitter: "Zuul (22348)"
Branch: master

commit 239ec3826a74c4f3ffb1239cc574f95c6097c631
Author: Jeremy Stanley <email address hidden>
Date: Thu Jul 8 20:49:35 2021 +0000

    Add OSSA-2021-001 (CVE-2021-20267)

    Change-Id: I6bcc8392831efbdc7759b0ed5340023bb0440c85
    Closes-Bug: #1902917

[Jeremy Stanley (fungi)]

The advisory is published at https://security.openstack.org/ now and copies have been sent to the usual mailing lists (oss-security, openstack-announce, openstack-discuss).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.