qrouter ns leak while last service port delete because of router gw port

Bug #1957794 reported by Krzysztof Tomaszewski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Krzysztof Tomaszewski

Bug Description

While removing last port from the subnet on compute host with DVR then L3 agent is cleaning unneeded qrouter-* namespaces.

When you have a different (even other user) VM on the same host that has port
from the subnet that your router has a gateway then deleting of qrouter namespaces is not triggered.

Scenario to reproduce:

Two instances multinode devstack master; no dhcp agent (for simplicity);
devstack default DVR router preconfiguration (public net as a default GW, private net as a subnet);
two nodes:
 - devstack1 - dvr_snat node,
 - devstack2 - dvr node

1) create a VM with private network on devstack2 node as a demo user:

(demo)$ openstack server create --net private --flavor cirros256 --image cirros-0.5.2-x86_64-disk test_private
(demo)$ openstack server show test_private -c id
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| id | 7e5bebfd-636d-4416-b2ce-7f16a7b720ca |
+-------+--------------------------------------+
(demo)$ openstack port list --device-id 7e5bebfd-636d-4416-b2ce-7f16a7b720ca -c id
+--------------------------------------+
| ID |
+--------------------------------------+
| d359efe3-8075-483a-90ee-807595d8786a |
+--------------------------------------+

There is proper tap interface and L3 agent creates qrouter-* namespace:

stack@devstack2:~/$ sudo ip netns | grep qr
qrouter-0a5fc7cf-0ed9-4fb9-921b-4ed95ef3924b (id: 0)
stack@devstack2:~/$ ip a | grep d359
28: tapd359efe3-80: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
stack@devstack2:~$ sudo ovs-vsctl get port tapd359efe3-80 tag
4
stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
name tag
-------------- ---
qr-c3ae7e60-aa 4
qr-7f7c0893-f7 4
tapd359efe3-80 4

2) create a VM with public network on devstack2 node as an admin user:

(admin)$ openstack server create --net public --flavor cirros256 --image cirros-0.5.2-x86_64-disk test_public
(admin)$ openstack server show test_public -c OS-EXT-SRV-ATTR:host -c id -c OS-EXT-STS:power_state -c OS-EXT-STS:vm_state
+------------------------+--------------------------------------+
| Field | Value |
+------------------------+--------------------------------------+
| OS-EXT-SRV-ATTR:host | devstack2 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:vm_state | active |
| id | 0622fd62-bb3e-4d36-bbcd-d0c8f8b14cc9 |
+------------------------+--------------------------------------+
(admin)$ openstack port list --device-id 0622fd62-bb3e-4d36-bbcd-d0c8f8b14cc9 -c id
+--------------------------------------+
| ID |
+--------------------------------------+
| dc822c75-715e-4788-9589-3fff05ccc307 |
+--------------------------------------+

stack@devstack2:~$ ip a | grep dc8
14: tapdc822c75-71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000

3) delete demo user test_private VM

(demo)$ openstack server delete test_private

VM is deleted but qrouter-* namespaces stays.

One VM only exist (admin's one):
stack@devstack2:~$ sudo virsh list --all
 Id Name State
-----------------------------------
 2 instance-00000007 running

stack@devstack2:~$ sudo ip netns | grep qr
qrouter-0a5fc7cf-0ed9-4fb9-921b-4ed95ef3924b (id: 0)
stack@devstack2:~$
stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
name tag
-------------- ---
qr-c3ae7e60-aa 4
qr-7f7c0893-f7 4

To clear this namespace you need to full resync L3 agent by agent restart or by disabling/enabling agent:

(admin)$ openstack network agent list --host devstack2 --agent-type l3 -c ID -c Host
+--------------------------------------+-----------+
| ID | Host |
+--------------------------------------+-----------+
| 77b01aa0-de3b-4b6b-a40a-08031460a97f | devstack2 |
+--------------------------------------+-----------+

(admin)$ openstack network agent set --disable 77b01aa0-de3b-4b6b-a40a-08031460a97f
(admin)$ openstack network agent set --enable 77b01aa0-de3b-4b6b-a40a-08031460a97f

and qrouter-* namespace disappear:

stack@devstack2:~$ sudo ip netns | grep qr
stack@devstack2:~$ sudo ovs-vsctl --format=table --columns=name,tag find port tag=4
name tag
---- ---
stack@devstack2:~$

Revision history for this message
Krzysztof Tomaszewski (labedz) wrote :
Changed in neutron:
assignee: nobody → Krzysztof Tomaszewski (labedz)
description: updated
description: updated
description: updated
Changed in neutron:
status: New → Confirmed
importance: Undecided → Medium
yatin (yatinkarel)
Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/824008
Committed: https://opendev.org/openstack/neutron/commit/4c6c48956ee8d9458924334f6650ca29e753a1ef
Submitter: "Zuul (22348)"
Branch: master

commit 4c6c48956ee8d9458924334f6650ca29e753a1ef
Author: labedz <email address hidden>
Date: Mon Jan 10 17:09:25 2022 +0100

    Exclude router gw subnet ports from port list while port delete.

    During port delete Neutron checks if host still handle some ports
    from subnets that are used by router.

    It may happen that the same subnet that is used for router gateway
    port may be used for another VM port. Then we skip notification
    to update L3 agent about related to router port deletion.

    Related-Bug: #1957794
    Change-Id: Iacc7d4e543ac261a6f566b49834b6865f6eded95

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/825076

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/825077

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/825076
Committed: https://opendev.org/openstack/neutron/commit/488a68cb49d3c076113aefd77b82d2c3374602ea
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 488a68cb49d3c076113aefd77b82d2c3374602ea
Author: labedz <email address hidden>
Date: Mon Jan 10 17:09:25 2022 +0100

    Exclude router gw subnet ports from port list while port delete.

    During port delete Neutron checks if host still handle some ports
    from subnets that are used by router.

    It may happen that the same subnet that is used for router gateway
    port may be used for another VM port. Then we skip notification
    to update L3 agent about related to router port deletion.

    Related-Bug: #1957794
    Change-Id: Iacc7d4e543ac261a6f566b49834b6865f6eded95
    (cherry picked from commit 4c6c48956ee8d9458924334f6650ca29e753a1ef)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/825077
Committed: https://opendev.org/openstack/neutron/commit/224332dbb522a47703ba7c0512a315dae70593b8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 224332dbb522a47703ba7c0512a315dae70593b8
Author: labedz <email address hidden>
Date: Mon Jan 10 17:09:25 2022 +0100

    Exclude router gw subnet ports from port list while port delete.

    During port delete Neutron checks if host still handle some ports
    from subnets that are used by router.

    It may happen that the same subnet that is used for router gateway
    port may be used for another VM port. Then we skip notification
    to update L3 agent about related to router port deletion.

    Related-Bug: #1957794
    Change-Id: Iacc7d4e543ac261a6f566b49834b6865f6eded95
    (cherry picked from commit 4c6c48956ee8d9458924334f6650ca29e753a1ef)

tags: added: in-stable-wallaby
Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.