Fuel for OpenStack

Comment 0 for bug 1401870

Revision history for this message

Oleg S. Gelbukh (gelbuhos) wrote on 2014-12-12:

This parameter is used by nova-network to determine if a bridge and gateway ip address of a network must be deleted from Compute node when the network is deleted in API. By default, it is set to 'False' which means that bridge, ip address and dnsmasq process listening on that address are left behind even after the network is deleted.

This leads to problems when another network with the same parameters (i.e. cidr and vlan id) created on that host: nova-network fails to start dnsmasq for that network.

```2014-12-12 10:49:58.062 8450 ERROR oslo.messaging.rpc.dispatcher [req-ba613287-419b-4ec3-8e06-b0eb30b93362 ] Exception during message handling: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"] NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-fi
le= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-h
ostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br203.hosts
Exit code: 2
Stdout: u''
Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 119, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 515, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher requested_networks=requested_networks)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 216, in _allocate_fixed_ips
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher vpn=vpn, address=address)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1778, in allocate_fixed_ip
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher self._setup_network_on_host(context, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1892, in _setup_network_on_host
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher self.driver.update_dhcp(elevated, dev, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher restart_dhcp(context, dev, network_ref)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher _execute(*cmd, run_as_root=True)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return utils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 195, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher cmd=sanitized_cmd)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"]
NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,
10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/va
r/lib/nova/networks/nova-br203.hosts
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Exit code: 2
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher
```

This leads to problems when another network with the same parameters (i.e. cidr and vlan id) created on that host: nova-network fails to start dnsmasq for that network.

```2014-12-12 10:49:58.062 8450 ERROR oslo.messaging.rpc.dispatcher [req-ba613287-419b-4ec3-8e06-b0eb30b93362 ] Exception during message handling: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"] NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-fi
le= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-h
ostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br203.hosts
Exit code: 2
Stdout: u''
Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 119, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 515, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     requested_networks=requested_networks)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 216, in _allocate_fixed_ips
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     vpn=vpn, address=address)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1778, in allocate_fixed_ip
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     self._setup_network_on_host(context, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1892, in _setup_network_on_host
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     self.driver.update_dhcp(elevated, dev, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     restart_dhcp(context, dev, network_ref)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     _execute(*cmd, run_as_root=True)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     return utils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 195, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher     cmd=sanitized_cmd)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"]
 NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,
10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/va
r/lib/nova/networks/nova-br203.hosts
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Exit code: 2
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher
```