nova-network DHCP server not correct when creating multiple networks

[Edmund Rhudy]

This bug pertains to stable Kilo, package versions below:
ii nova-api 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-cert 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - certificate management
ii nova-common 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii nova-conductor 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
ii nova-network 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - Network manager
ii nova-novncproxy 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-nova-adminclient 0.1.8-0ubuntu2 amd64 client for administering Openstack Nova
ii python-novaclient 1:2.22.0-0ubuntu1~cloud0 all client library for OpenStack Compute API

This bug was originally reported at https://github.com/bloomberg/chef-bcpc/issues/573. I triaged it there and am distilling it down here.

In essence, we have a command in a Chef recipe that builds out a number of fixed IP networks using nova-manage during initial setup of an OpenStack cluster. The command looks like this:

nova-manage network create --label fixed --fixed_range_v4=1.104.0.0/16 --num_networks=16 --multi_host=T --network_size=128 --vlan_start=1000 --bridge_interface=p2p2

As per the original bug report on GitHub, all subnets after the first one were being created with a DHCP server address of the gateway of the first subnet to be created. I dug in and found the problem code in nova/network/manager.py; if dhcp_server is not provided, the first iteration through the subnets enumeration will set it to the gateway IP. Since dhcp_server is scoped at method level, it sticks around for the entire loop and so every created subnet gets the same DHCP server IP. A little println debugging indicated that this was indeed the case.

This causes things to break for us pretty badly. When launching instances in any network other than the first one, launch will fail because Nova tries to launch dnsmasq and bind it to an IP that's already bound by another instance of dnsmasq, which fails.

I patched manager.py in the following way, which writes the DHCP server IP to a local variable in the loop. Specifying a DHCP server manually will still override inferring the DHCP server from the gateway address. Tests pass after this change (sorry that it is not in Gerrit already, but I haven't been able to sit down and get git review working yet):

diff --git a/nova/network/manager.py b/nova/network/manager.py
index 3e8e8b1..832fd1b 100644
--- a/nova/network/manager.py
+++ b/nova/network/manager.py
@@ -1351,13 +1351,15 @@ class NetworkManager(manager.Manager):
                 else:
                     net.gateway = current
                     current += 1
- if not dhcp_server:
- dhcp_server = net.gateway
+ if dhcp_server:
+ subnet_dhcp_server = dhcp_server
+ else:
+ subnet_dhcp_server = net.gateway
                 net.dhcp_start = current
                 current += 1
- if str(net.dhcp_start) == dhcp_server:
+ if str(net.dhcp_start) == subnet_dhcp_server:
                     net.dhcp_start = current
- net.dhcp_server = dhcp_server
+ net.dhcp_server = subnet_dhcp_server
                 extra_reserved.append(str(net.dhcp_server))
                 extra_reserved.append(str(net.gateway))

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/184620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/184787

[OpenStack Infra (hudson-openstack)]

Change abandoned by Edmund Rhudy (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/184620

[OpenStack Infra (hudson-openstack)]

Change abandoned by Edmund Rhudy (<email address hidden>) on branch: master
Review: https://review.openstack.org/184787