Neutron server doesn't wait for port DHCP provisioning while VM creation

[Anton Kurbatov]

I found that neutron-server does not wait for successful port provisioning from the dhcp agent in the case of VM creation. DHCP entity is not added into provisioning_block by neutron-server for such port.
As a result, nova receives a notification that the port is plugged, while the DHCP agent is still processing the port or even getting an error during processing.

Steps to reproduce on devstack from master:

- make port_create_end method fail in DHCP agent side [1]
- create a VM with network with DHCP enabled

VM is successfully created, port is active, while the DHCP entry for this port is not configured.

[root@node0 neutron]# git diff
diff --git a/neutron/agent/dhcp/agent.py b/neutron/agent/dhcp/agent.py
index 7349d7e297..553ba81fdc 100644
--- a/neutron/agent/dhcp/agent.py
+++ b/neutron/agent/dhcp/agent.py
@@ -676,6 +676,7 @@ class DhcpAgent(manager.Manager):
                                     payload.get('priority', DEFAULT_PRIORITY),
                                     action='_port_create',
                                     resource=created_port, obj_type='port')
+ raise Exception('fail for testing purposes')
         self._queue.add(update)

     @_wait_if_syncing

[root@node0 neutron]# openstack server create test-vm --network net1 --flavor m1.tiny --image cirros-0.5.2-x86_64-disk
[root@node0 ~]# openstack server list
+--------------------------------------+---------+--------+--------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------+--------+--------------------+--------------------------+---------+
| cce75084-b1e0-4407-a0d6-0074ed05abad | test-vm | ACTIVE | net1=192.168.1.111 | cirros-0.5.2-x86_64-disk | m1.tiny |
+--------------------------------------+---------+--------+--------------------+--------------------------+---------+
[root@node0 ~]# openstack port list --device-id cce75084-b1e0-4407-a0d6-0074ed05abad
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| d7e55e08-05ae-4ac4-8cd0-4f88b93c5872 | | fa:16:3e:9e:30:b3 | ip_address='192.168.1.111', subnet_id='281f70f3-8996-436b-ab90-bff1f9dbf5f8' | ACTIVE |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
[root@node0 ~]#
[root@node0 ~]# cat /opt/stack/data/neutron/dhcp/710bcfcd-44d9-445d-a895-8ec522f64016/addn_hosts
[root@node0 ~]#

While VM creation there two API calls from the nova:
1) Port 'create' API call:
Nov 22 16:19:40 node0 neutron-server[953593]: DEBUG neutron.api.v2.base [req-5cbe6387-fe21-4509-81f6-cfcfe268252f req-0b7496ea-3697-4bc8-abb4-95d8f23d3497 demo admin] Request body: {'port': {'device_id': 'cce75084-b1e0-4407-a0d6-0074ed05abad', 'network_id': '710bcfcd-44d9-445d-a895-8ec522f64016', 'admin_state_up': True, 'tenant_id': 'a022c969871149e9b19ec31c896a0701'}} {{(pid=953593) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:730}}

2) Port 'update' API call:
Nov 22 16:16:11 node0 neutron-server[953593]: DEBUG neutron.api.v2.base [req-145264e0-96a0-450b-9ad5-a5181c2497b1 req-9015e2c3-7dbb-430f-9cba-c7d6972f5134 service neutron] Request body: {'port': {'device_id': '4a4f87c0-a357-49eb-8639-58b499b8ae1f', 'device_owner': 'compute:nova', 'binding:host_id': 'node1'}} {{(pid=953593) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:730}}

For the port creation API call a DHCP provisioning is not setup because device_owner is absent [2]
For the port 'update' API call a DCHP provisioning is not setup because none of the fixed_ips/mac_address is updated [3]

[1] https://opendev.org/openstack/neutron/src/commit/51827d8e78db4926f3aa347c4b2237a7b210f861/neutron/agent/dhcp/agent.py#L670
[2] https://opendev.org/openstack/neutron/src/commit/51827d8e78db4926f3aa347c4b2237a7b210f861/neutron/plugins/ml2/plugin.py#L1501
[3] https://opendev.org/openstack/neutron/src/commit/51827d8e78db4926f3aa347c4b2237a7b210f861/neutron/plugins/ml2/plugin.py#L1925

[Lajos Katona (lajos-katona)]

Hi, so you suggest if I understand well to wait for provisioning in case of update of the port, for example by checking if the device_owner was set during the update?

[Anton Kurbatov (akurbatov)]

Hi Lajos,
If only the device_owner at the port changes and the neutron-server will set the provisioning block, then this will not be very logical, because from the point of view of the dhcp agent, nothing has changed from what needs to be changed in the dnsmasq configs.
In addition, in such a case we may again meet the bug #1982367 and related fix [1] because we will have to force this port to process on the dhcp agent side.
In my opinion more reasonable to set the block provisioning for DHCP if the device_owner is equal to an empty string when creating a port, something like this:

diff --git a/neutron/plugins/ml2/plugin.py b/neutron/plugins/ml2/plugin.py
index e52071aee6..9174a6edfe 100644
--- a/neutron/plugins/ml2/plugin.py
+++ b/neutron/plugins/ml2/plugin.py
@@ -1499,7 +1499,8 @@ class Ml2Plugin(db_base_plugin_v2.NeutronDbPluginV2,
         if not cfg.CONF.enable_traditional_dhcp:
             return

- if port['device_owner'] in ml2_consts.NO_PBLOCKS_TYPES:
+ if (port['device_owner'] in ml2_consts.NO_PBLOCKS_TYPES and
+ port['device_owner'] != ml2_consts.DEFAULT_DEVICE_OWNER):
             # do not set provisioning_block if it is neutron service port
             return

[1] https://opendev.org/openstack/neutron/commit/06ddcaf4368ce054a8d396e70880f7dd1abe0562

[Anton Kurbatov (akurbatov)]

Easier way to reproduce: stop Neutron DHCP agent(s) and create a VM in the network with DHCP enabled. The VM is successfully created.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/865470

[Oleg Bondarev (obondarev)]

Logically Nova should create port with proper device owner. I don't know why it's not like that now, was it always this way?