DVR qrouters are not created when VMs are added after the router-interface is added to the router

[Swaminathan Vasudevan]

qrouters for DVR routers should be created on demand when VMs are created on the compute Node.

But with the current code, it seems that is broken.

When a VM is created on a router's subnet after the router-interface is added to the router, then the 'qrouter' namespace is not created on the compute Node. It is only created on the "dvr_snat" node and not on the "dvr" node.

stack@ubuntu-multinet-ctlr:~/devstack$ neutron net-create net1
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | ccaeaf39-4c33-40b8-b0e0-551414a86ca3 |
| name | net1 |
| provider:network_type | vxlan |
| provider:physical_network | |
| provider:segmentation_id | 1003 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
+---------------------------+--------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ neutron subnet-create net1 --name subnet1 10.20.0.0/24
Created a new subnet:
+-------------------+----------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------+
| allocation_pools | {"start": "10.20.0.2", "end": "10.20.0.254"} |
| cidr | 10.20.0.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.20.0.1 |
| host_routes | |
| id | e404b9cb-dd2d-4919-8b6c-72799ae7efed |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | subnet1 |
| network_id | ccaeaf39-4c33-40b8-b0e0-551414a86ca3 |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
+-------------------+----------------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ neutron router-create router2
Created a new router:
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| distributed | True |
| external_gateway_info | |
| ha | False |
| id | e82022cd-d45c-4558-9d35-86a1e5f58462 |
| name | router2 |
| routes | |
| status | ACTIVE |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
+-----------------------+--------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ neutron router-interface-add router2 subnet1
Added interface f872d6ea-f2c9-4334-8090-48d9542cdbef to router router2.
stack@ubuntu-multinet-ctlr:~/devstack$ neutron router-gateway-set router2 publicSet gateway for router router2
stack@ubuntu-multinet-ctlr:~/devstack$
stack@ubuntu-multinet-ctlr:~/devstack$
stack@ubuntu-multinet-ctlr:~/devstack$ sudo ip netns list
qrouter-e82022cd-d45c-4558-9d35-86a1e5f58462
qdhcp-5f413f3e-573f-404d-9d24-b7b11b141278
snat-acfc7720-3071-46a3-8be3-1b9430ddb47e
qrouter-acfc7720-3071-46a3-8be3-1b9430ddb47e
stack@ubuntu-multinet-ctlr:~/devstack$ sudo ip netns list
snat-e82022cd-d45c-4558-9d35-86a1e5f58462
qrouter-e82022cd-d45c-4558-9d35-86a1e5f58462
qdhcp-5f413f3e-573f-404d-9d24-b7b11b141278
snat-acfc7720-3071-46a3-8be3-1b9430ddb47e
qrouter-acfc7720-3071-46a3-8be3-1b9430ddb47e
stack@ubuntu-multinet-ctlr:~/devstack$ sudo ip netns list
snat-e82022cd-d45c-4558-9d35-86a1e5f58462
qrouter-e82022cd-d45c-4558-9d35-86a1e5f58462
qdhcp-5f413f3e-573f-404d-9d24-b7b11b141278
snat-acfc7720-3071-46a3-8be3-1b9430ddb47e
qrouter-acfc7720-3071-46a3-8be3-1b9430ddb47e
stack@ubuntu-multinet-ctlr:~/devstack$ nova boot --flavor 42 --image cirros-0.3.2-x86_64-uec --nic net-id=ccaeaf39-4c33-40b8-b0e0-551414a86ca3 mycirros-new1
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000002 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | yD5n5R7KMuMj |
| config_drive | |
| created | 2015-01-17T00:09:11Z |
| flavor | m1.nano (42) |
| hostId | |
| id | 53537468-7089-4abf-ab25-696af363c6c4 |
| image | cirros-0.3.2-x86_64-uec (98d3d4ad-065b-4601-b9ed-51fe21cea7db) |
| key_name | - |
| metadata | {} |
| name | mycirros-new1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
| updated | 2015-01-17T00:09:11Z |
| user_id | f19a68c6af61462f889df48fad5d4c36 |
+--------------------------------------+----------------------------------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ nova list
+--------------------------------------+---------------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+------------------+
| ebaf4276-ce74-47f3-aa5b-65ca81e30f61 | mycirros-new | ACTIVE | - | Running | private=10.0.0.3 |
| 53537468-7089-4abf-ab25-696af363c6c4 | mycirros-new1 | ACTIVE | - | Running | net1=10.20.0.3 |
+--------------------------------------+---------------+--------+------------+-------------+------------------+
stack@ubuntu-multinet-ctlr:~/devstack$
stack@ubuntu-multinet-ctlr:~/devstack$ sudo ip netns list
qdhcp-ccaeaf39-4c33-40b8-b0e0-551414a86ca3
snat-e82022cd-d45c-4558-9d35-86a1e5f58462
qrouter-e82022cd-d45c-4558-9d35-86a1e5f58462
qdhcp-5f413f3e-573f-404d-9d24-b7b11b141278
snat-acfc7720-3071-46a3-8be3-1b9430ddb47e
qrouter-acfc7720-3071-46a3-8be3-1b9430ddb47e
stack@ubuntu-multinet-ctlr:~/devstack$ neutron l3-agent-hosting-router router2
Unknown command [u'l3-agent-hosting-router', u'router2']
stack@ubuntu-multinet-ctlr:~/devstack$ neutron l3-agent-list-hosting-router router2
+--------------------------------------+----------------------+----------------+-------+
| id | host | admin_state_up | alive |
+--------------------------------------+----------------------+----------------+-------+
| ef0fcc03-dfb3-4b29-a78f-f278ffe83f4c | ubuntu-multinet-ctlr | True | :-) |
+--------------------------------------+----------------------+----------------+-------+
stack@ubuntu-multinet-ctlr:~/devstack$
stack@ubuntu-multinet-ctlr:~/devstack$ neutron agent-list
+--------------------------------------+--------------------+-------------------------+-------+----------------+---------------------------+
| id | agent_type | host | alive | admin_state_up | binary |
+--------------------------------------+--------------------+-------------------------+-------+----------------+---------------------------+
| 3d330111-58db-4480-a7bd-eb2ee7447af7 | Metadata agent | ubuntu-multinet-ctlr | :-) | True | neutron-metadata-agent |
| 80aa9d84-9f06-4187-a925-b0c77bab68c6 | L3 agent | ubuntu-multinet-compute | :-) | True | neutron-l3-agent |
| 88aeed99-3942-44c6-a933-3cd85bf918d1 | Open vSwitch agent | ubuntu-multinet-ctlr | :-) | True | neutron-openvswitch-agent |
| 957a87bb-2972-4315-a260-1e5068f93451 | Metadata agent | ubuntu-multinet-compute | :-) | True | neutron-metadata-agent |
| b0d98f4f-a0a1-42d6-b6f4-3b18e142861e | DHCP agent | ubuntu-multinet-ctlr | :-) | True | neutron-dhcp-agent |
| dfa7b986-6f3d-4bac-b111-61a4cc3a811b | Open vSwitch agent | ubuntu-multinet-compute | :-) | True | neutron-openvswitch-agent |
| ef0fcc03-dfb3-4b29-a78f-f278ffe83f4c | L3 agent | ubuntu-multinet-ctlr | :-) | True | neutron-l3-agent |
+--------------------------------------+--------------------+-------------------------+-------+----------------+---------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$
stack@ubuntu-multinet-ctlr:~/devstack$ sudo ip netns list
qdhcp-ccaeaf39-4c33-40b8-b0e0-551414a86ca3
snat-e82022cd-d45c-4558-9d35-86a1e5f58462
qrouter-e82022cd-d45c-4558-9d35-86a1e5f58462
qdhcp-5f413f3e-573f-404d-9d24-b7b11b141278
snat-acfc7720-3071-46a3-8be3-1b9430ddb47e
qrouter-acfc7720-3071-46a3-8be3-1b9430ddb47e
stack@ubuntu-multinet-ctlr:~/devstack$ neutron net-create net2
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 9942caf3-0bbe-4fc0-80c6-72f663d017db |
| name | net2 |
| provider:network_type | vxlan |
| provider:physical_network | |
| provider:segmentation_id | 1004 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
+---------------------------+--------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ neutron subnet-create net2 --name subnet2 10.30.0.0/24
Created a new subnet:
+-------------------+----------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------+
| allocation_pools | {"start": "10.30.0.2", "end": "10.30.0.254"} |
| cidr | 10.30.0.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.30.0.1 |
| host_routes | |
| id | c0f1990a-2845-43e0-95e4-a6b52c81b62b |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | subnet2 |
| network_id | 9942caf3-0bbe-4fc0-80c6-72f663d017db |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
+-------------------+----------------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ nova boot --flavor 42 --image cirros-0.3.2-x86_64-uec --nic net-id=9942caf3-0bbe-4fc0-80c6-72f663d017db mycirros-new2
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | ZTfGdgJaFz9X |
| config_drive | |
| created | 2015-01-17T00:17:06Z |
| flavor | m1.nano (42) |
| hostId | |
| id | b8e66998-97e9-4c4c-b64d-b97847fdb41f |
| image | cirros-0.3.2-x86_64-uec (98d3d4ad-065b-4601-b9ed-51fe21cea7db) |
| key_name | - |
| metadata | {} |
| name | mycirros-new2 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
| updated | 2015-01-17T00:17:06Z |
| user_id | f19a68c6af61462f889df48fad5d4c36 |
+--------------------------------------+----------------------------------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ nova list
+--------------------------------------+---------------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+------------------+
| ebaf4276-ce74-47f3-aa5b-65ca81e30f61 | mycirros-new | ACTIVE | - | Running | private=10.0.0.3 |
| 53537468-7089-4abf-ab25-696af363c6c4 | mycirros-new1 | ACTIVE | - | Running | net1=10.20.0.3 |
| b8e66998-97e9-4c4c-b64d-b97847fdb41f | mycirros-new2 | ACTIVE | - | Running | net2=10.30.0.2 |
+--------------------------------------+---------------+--------+------------+-------------+------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ neutron router-create router3
Created a new router:
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| distributed | True |
| external_gateway_info | |
| ha | False |
| id | 7bacb095-134f-4a38-846b-ce20348def3e |
| name | router3 |
| routes | |
| status | ACTIVE |
| tenant_id | f822fe0773cd4d21bddb4ecb2477f21d |
+-----------------------+--------------------------------------+
stack@ubuntu-multinet-ctlr:~/devstack$ neutron router-interface-add router3 subnet2
Added interface 934d2899-d912-4b4f-926b-0fc1320a9aac to router router3.
stack@ubuntu-multinet-ctlr:~/devstack$
stack@ubuntu-multinet-ctlr:~/devstack$ neutron l3-agent-list-hosting-router router3
+--------------------------------------+-------------------------+----------------+-------+
| id | host | admin_state_up | alive |
+--------------------------------------+-------------------------+----------------+-------+
| 80aa9d84-9f06-4187-a925-b0c77bab68c6 | ubuntu-multinet-compute | True | :-) |
| ef0fcc03-dfb3-4b29-a78f-f278ffe83f4c | ubuntu-multinet-ctlr | True | :-) |
+--------------------------------------+-------------------------+----------------+-------+
stack@ubuntu-multinet-ctlr:~/devstack$ sudo ip netns list
qrouter-7bacb095-134f-4a38-846b-ce20348def3e
qdhcp-9942caf3-0bbe-4fc0-80c6-72f663d017db
qdhcp-ccaeaf39-4c33-40b8-b0e0-551414a86ca3
snat-e82022cd-d45c-4558-9d35-86a1e5f58462
qrouter-e82022cd-d45c-4558-9d35-86a1e5f58462
qdhcp-5f413f3e-573f-404d-9d24-b7b11b141278
snat-acfc7720-3071-46a3-8be3-1b9430ddb47e
qrouter-acfc7720-3071-46a3-8be3-1b9430ddb47e
stack@ubuntu-multinet-ctlr:~/devstack$ neutron router-gateway-set router3 publicSet gateway for router router3
stack@ubuntu-multinet-ctlr:~/devstack$
stack@ubuntu-multinet-ctlr:~/devstack$ sudo ip netns list
snat-7bacb095-134f-4a38-846b-ce20348def3e
qrouter-7bacb095-134f-4a38-846b-ce20348def3e
qdhcp-9942caf3-0bbe-4fc0-80c6-72f663d017db
qdhcp-ccaeaf39-4c33-40b8-b0e0-551414a86ca3
snat-e82022cd-d45c-4558-9d35-86a1e5f58462
qrouter-e82022cd-d45c-4558-9d35-86a1e5f58462
qdhcp-5f413f3e-573f-404d-9d24-b7b11b141278
snat-acfc7720-3071-46a3-8be3-1b9430ddb47e
qrouter-acfc7720-3071-46a3-8be3-1b9430ddb47e
stack@ubuntu-multinet-ctlr:~/devstack$

[Swaminathan Vasudevan (swaminathan-vasudevan)]

hi yalei wang, do you have a fix for this right now.
I think I found out the problem.
Please push in your patch if you already have one and I can review it, otherwise will it be ok, if can take over this bug.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

The latest patch in "ml2' have caused this issue.

Move postcommit ops out of transaction for bulk
Change-Id: I9a9683058088e50d9443040223232bf5e1396ccf
Closes-Bug: #1193861

This re-orders the call between the "_get_host_port_if_changed" and "_process_bind_router".

That is the reason that "_notify_l3_agent_new_port" is not called.

Do we think that we need to revert the "above" patch ????

[Carl Baldwin (carl-baldwin)]

The code needs to remember what "mech_context._binding.host” was before calling _process_bind_router (e.g. “old_host") and then compare that to “host” (e.g. “old_host” == “host”).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/149143

[ZongKai LI (zongkai)]

This case will also happened in multi compute nodes env with DVR enabled.
If we check db, we can find port binding created, new created instance's port and the instance's host.
But no netns qrouter-ROUTERID is created on that host.

[Assaf Muller (amuller)]

I think that his is a blocker / critical bug for DVR. It would be a shame to release Kilo without a solution.

[Kyle Mestery (mestery)]

According to Assaf in comment #6, this looks like a release blocker. Raising the priority a bit and marking as Kilo-RC1 to gain this visibility. Swami, is this still planned for Kilo? I don't see a patch and we'd need one this week.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Hi Kyle, yes there is a patch to address this problem.

https://review.openstack.org/#/c/155317/

need reviewers for this patch.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I did some testing today from top of the branch and I don't see this behavior anymore.
Hopefully the patch added by "Armando" to fix the "register_notify" function for create_port, delete_port and update_port might have fixed this problem.

You can go ahead and close this bug.

[Salvatore Orlando (salvatore-orlando)]

If we can close the bug, does this also mean there is no need to merge patch #155317?

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Patch #155317 can still be merged, but it is no more associated with the current bug #1411883

[Armando Migliaccio (armando-migliaccio)]

Why am I quoted? Am I am not real? :)

[Armando Migliaccio (armando-migliaccio)]

This looks an awful lot like https://bugs.launchpad.net/neutron/+bug/1438969, but I am bit puzzled from Swami's statement #19. I merely (and hopefully) refactored the code, with no intention to alter pre-existing the logic, buggy or otherwise.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Armando your refactored code helped to fix this problem.
In the previous version, there was check on the _get_host_port_if_changed return value to make a call to the "_notify_l3_agent_".

But now with the register_notify changes from your patch, that check is not in place and so the L3-agents are notified.

[Assaf Muller (amuller)]

I can confirm that I played around with DVR a couple of weeks ago and hit this bug, and yesterday I did not. I created a new VM and a qrouter namespace appeared on the VM's host.

[Armando Migliaccio (armando-migliaccio)]

This can no longer be reproduced.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/149143
Reason: bug no longer reproducible

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Yes this bug can be closed. This bug was exposed and was latter not reproducable.