neutron

Neutron allows you to delete router_ha_interface ports, which can lead to issues

Bug #2008270 reported by Anton Kurbatov on 2023-02-23

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Medium	Anton Kurbatov

Bug Description

We ran into a problem with a customer when some external integration tries to remove all ports using the neutron API, including router prots.

It seems only the router ports with the router_ha_interface device owner are allowed to delete, all other router ports cannot be deleted directly through the API.

Here is a simple example that demonstrates the doubling of ARP responses if such a port is deleted:

[root@dev0 ~]# openstack router create r1 --ha --external-gateway public -c id
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| id | 5d9d6fee-6652-4843-9f7c-54c11899d721 |
+-------+--------------------------------------+
[root@dev0 ~]# neutron l3-agent-list-hosting-router r1
neutron CLI is deprecated and will be removed in the Z cycle. Use openstack CLI instead.
+--------------------------------------+------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+------+----------------+-------+----------+
| 9dd0920a-cb0c-47f1-a976-3e208e3e2e6c | dev0 | True | :-) | active |
| 6fa92056-ca25-42e0-aee4-c4e744008239 | dev2 | True | :-) | standby |
| 8fbda128-dc9c-4b3b-be1b-bb3f11ad1447 | dev1 | True | :-) | standby |
+--------------------------------------+------+----------------+-------+----------+
[root@dev0 ~]# openstack port list --device-id 5d9d6fee-6652-4843-9f7c-54c11899d721 -c id -c device_owner -c fixed_ips --long
+--------------------------------------+-----------------------------+--------------------------------------------------------------------------------+
| ID | Device Owner | Fixed IP Addresses |
+--------------------------------------+-----------------------------+--------------------------------------------------------------------------------+
| 555a9272-c9df-4a05-9f08-752c91c5a4c9 | network:router_ha_interface | ip_address='169.254.192.147', subnet_id='20c159f7-13f8-4093-9a4a-8380bdcfea60' |
| 6a196ff7-f3d4-4bee-aed0-b5d7ba727741 | network:router_ha_interface | ip_address='169.254.193.243', subnet_id='20c159f7-13f8-4093-9a4a-8380bdcfea60' |
| 7a849dcc-eac4-4d5b-a547-7ce3986ffb95 | network:router_ha_interface | ip_address='169.254.192.155', subnet_id='20c159f7-13f8-4093-9a4a-8380bdcfea60' |
| d77e624d-87a2-4135-9118-3d8e78539cee | network:router_gateway | ip_address='10.136.17.172', subnet_id='ee15c548-e497-449e-b46d-50e9ccc0f70c' |
+--------------------------------------+-----------------------------+--------------------------------------------------------------------------------+
[root@dev0 ~]#

[root@dev0 ~]# ip netns exec snat-5d9d6fee-6652-4843-9f7c-54c11899d721 ip a
...
25: ha-555a9272-c9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:7d:cf:a0 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.147/18 brd 169.254.255.255 scope global ha-555a9272-c9
       valid_lft forever preferred_lft forever
    inet 169.254.0.189/24 scope global ha-555a9272-c9
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe7d:cfa0/64 scope link
       valid_lft forever preferred_lft forever
28: qg-d77e624d-87: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:a8:54:29 brd ff:ff:ff:ff:ff:ff
    inet 10.136.17.172/20 scope global qg-d77e624d-87
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea8:5429/64 scope link nodad
       valid_lft forever preferred_lft forever
[root@dev0 ~]#

[root@dev0 ~]# ip netns exec snat-5d9d6fee-6652-4843-9f7c-54c11899d721 ip a s qg-d77e624d-87
28: qg-d77e624d-87: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:a8:54:29 brd ff:ff:ff:ff:ff:ff
    inet 10.136.17.172/20 scope global qg-d77e624d-87
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea8:5429/64 scope link nodad
       valid_lft forever preferred_lft forever
[root@dev0 ~]# ssh dev2 ip netns exec snat-5d9d6fee-6652-4843-9f7c-54c11899d721 ip a s qg-d77e624d-87
28: qg-d77e624d-87: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:a8:54:29 brd ff:ff:ff:ff:ff:ff
    inet 10.136.17.172/20 scope global qg-d77e624d-87
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea8:5429/64 scope link nodad
       valid_lft forever preferred_lft forever
[root@dev0 ~]#

[root@dev0 ~]# arping -c 1 -I eth0 10.136.17.172
ARPING 10.136.17.172 from 10.136.20.188 eth0
Unicast reply from 10.136.17.172 [FA:16:3E:A8:54:29] 1.537ms
Unicast reply from 10.136.17.172 [FA:16:3E:A8:54:29] 2.383ms
Sent 1 probes (1 broadcast(s))
Received 2 response(s)
[root@dev0 ~]#

As you can see, after deleting the HA port, we got a doubling of the ARP responses, which can lead to further problems in the roiting.

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-23: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874931

Changed in neutron:
status:	New → In Progress

Bernard Cafarelli (bcafarel) on 2023-02-27

tags:	added: l3-ha
Changed in neutron:
assignee:	nobody → Anton Kurbatov (akurbatov)
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-28: Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874931
Committed: https://opendev.org/openstack/neutron/commit/e68e4162cebfa8e5d8f70d4220119d5ca54666bf
Submitter: "Zuul (22348)"
Branch: master

commit e68e4162cebfa8e5d8f70d4220119d5ca54666bf
Author: Anton Kurbatov <email address hidden>
Date: Thu Feb 23 15:07:11 2023 +0000

Prevent router_ha_interface port from being removed via API

    If someone removes the port with device owner router_ha_interface,
    then we can get unexpected router behavior like doubling
    arp response packets. This patch prohibits removing such a port.

Closes-Bug: #2008270
Change-Id: Ief031801c1a3e3dd64e6cbf65e27f04f2bef9cba

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-28: Fix proposed to neutron (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/875737

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-28: Fix proposed to neutron (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/875738

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-28: Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/875739

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-01: Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/875820

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-01: Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875739
Committed: https://opendev.org/openstack/neutron/commit/7e7eff2d35b43e2a885dbf1eb18e8674945d8c96
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 7e7eff2d35b43e2a885dbf1eb18e8674945d8c96
Author: Anton Kurbatov <email address hidden>
Date: Thu Feb 23 15:07:11 2023 +0000

Prevent router_ha_interface port from being removed via API

    If someone removes the port with device owner router_ha_interface,
    then we can get unexpected router behavior like doubling
    arp response packets. This patch prohibits removing such a port.

    Closes-Bug: #2008270
    Change-Id: Ief031801c1a3e3dd64e6cbf65e27f04f2bef9cba
    (cherry picked from commit e68e4162cebfa8e5d8f70d4220119d5ca54666bf)

tags:	added: in-stable-xena
tags:	added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-01: Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875738
Committed: https://opendev.org/openstack/neutron/commit/1f9f77e4a27ac41084037df625fef124fb19163c
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 1f9f77e4a27ac41084037df625fef124fb19163c
Author: Anton Kurbatov <email address hidden>
Date: Thu Feb 23 15:07:11 2023 +0000

Prevent router_ha_interface port from being removed via API

    If someone removes the port with device owner router_ha_interface,
    then we can get unexpected router behavior like doubling
    arp response packets. This patch prohibits removing such a port.

    Closes-Bug: #2008270
    Change-Id: Ief031801c1a3e3dd64e6cbf65e27f04f2bef9cba
    (cherry picked from commit e68e4162cebfa8e5d8f70d4220119d5ca54666bf)

tags:

added: in-stable-zed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-01: Fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875737
Committed: https://opendev.org/openstack/neutron/commit/2913dae7a3108d1b2bc72e8d892da302f41ca8bc
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 2913dae7a3108d1b2bc72e8d892da302f41ca8bc
Author: Anton Kurbatov <email address hidden>
Date: Thu Feb 23 15:07:11 2023 +0000

Prevent router_ha_interface port from being removed via API

    If someone removes the port with device owner router_ha_interface,
    then we can get unexpected router behavior like doubling
    arp response packets. This patch prohibits removing such a port.

    Closes-Bug: #2008270
    Change-Id: Ief031801c1a3e3dd64e6cbf65e27f04f2bef9cba
    (cherry picked from commit e68e4162cebfa8e5d8f70d4220119d5ca54666bf)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-02: Fix included in openstack/neutron 22.0.0.0rc1

#10

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-08: Fix merged to neutron (stable/wallaby)

#11

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875820
Committed: https://opendev.org/openstack/neutron/commit/e9de8123d5c0f4a36227055d18690713018220a1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit e9de8123d5c0f4a36227055d18690713018220a1
Author: Anton Kurbatov <email address hidden>
Date: Thu Feb 23 15:07:11 2023 +0000

Prevent router_ha_interface port from being removed via API

    If someone removes the port with device owner router_ha_interface,
    then we can get unexpected router behavior like doubling
    arp response packets. This patch prohibits removing such a port.

Conflicts:
neutron/tests/unit/db/test_l3_hamode_db.py

    Closes-Bug: #2008270
    Change-Id: Ief031801c1a3e3dd64e6cbf65e27f04f2bef9cba
    (cherry picked from commit e68e4162cebfa8e5d8f70d4220119d5ca54666bf)