teardown_unused_network_gateway parameter is False by default

[Oleg S. Gelbukh]

This parameter is used by nova-network to determine if a bridge and gateway ip address of a network must be deleted from Compute node when the network is deleted in API. By default, it is set to 'False' which means that bridge, ip address and dnsmasq process listening on that address are left behind even after the network is deleted.

This leads to problems when another network with the same parameters (i.e. cidr and vlan id) created on that host: nova-network fails to start dnsmasq for that network.

2014-12-12 10:49:58.062 8450 ERROR oslo.messaging.rpc.dispatcher [req-ba613287-419b-4ec3-8e06-b0eb30b93362 ] Exception during message handling: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"] NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-fi
le= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-h
ostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br203.hosts
Exit code: 2
Stdout: u''
Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 119, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 515, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher requested_networks=requested_networks)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 216, in _allocate_fixed_ips
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher vpn=vpn, address=address)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1778, in allocate_fixed_ip
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher self._setup_network_on_host(context, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1892, in _setup_network_on_host
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher self.driver.update_dhcp(elevated, dev, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher restart_dhcp(context, dev, network_ref)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher _execute(*cmd, run_as_root=True)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return utils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 195, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher cmd=sanitized_cmd)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"]
 NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,
10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/va
r/lib/nova/networks/nova-br203.hosts
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Exit code: 2
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/151196

[Bogdan Dobrelya (bogdando)]

It looks like we should backport this medium bug for 6.0 and 5.1 as well

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/153493

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/153494

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151196
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6aa926d89540f4af1af1d58e55fe6eddacba6e58
Submitter: Jenkins
Branch: master

commit 6aa926d89540f4af1af1d58e55fe6eddacba6e58
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon Feb 2 11:00:55 2015 +0100

    Enable teardown_unused_network_gateway for nova-network deployments

    The parameter is used by nova-network to determine if a bridge and
    gateway IP address of a network must be deleted from the compute node
    when the network is deleted in API. By default, it is set to 'False'
    which means that bridge, IP address and dnsmasq process listening on
    that address are left behind even after the network is deleted.

    Change-Id: I4c94bb03501f4238ead2378cf504485b7d67b236
    Closes-Bug: 1401870

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/153494
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=59a2149566b07e11c4520fa9f885732ea1041e46
Submitter: Jenkins
Branch: stable/5.1

commit 59a2149566b07e11c4520fa9f885732ea1041e46
Author: Bartłomiej Piotrowski <email address hidden>
Date: Thu Jan 29 12:56:57 2015 +0100

    Enable teardown_unused_network_gateway for nova-network deployments

    The parameter is used by nova-network to determine if a bridge and
    gateway IP address of a network must be deleted from the compute node
    when the network is deleted in API. By default, it is set to 'False'
    which means that bridge, IP address and dnsmasq process listening on
    that address are left behind even after the network is deleted.

    Change-Id: I4c94bb03501f4238ead2378cf504485b7d67b236
    Closes-Bug: 1401870

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/153493
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7b9cbc30d457600649fdf57c25990493a075069f
Submitter: Jenkins
Branch: stable/6.0

commit 7b9cbc30d457600649fdf57c25990493a075069f
Author: Bartłomiej Piotrowski <email address hidden>
Date: Thu Jan 29 12:56:57 2015 +0100

    Enable teardown_unused_network_gateway for nova-network deployments

    The parameter is used by nova-network to determine if a bridge and
    gateway IP address of a network must be deleted from the compute node
    when the network is deleted in API. By default, it is set to 'False'
    which means that bridge, IP address and dnsmasq process listening on
    that address are left behind even after the network is deleted.

    Change-Id: I4c94bb03501f4238ead2378cf504485b7d67b236
    Closes-Bug: 1401870

[Bartłomiej Piotrowski (bpiotrowski)]

Verified on 6.1 #471.

[root@node-1 ~]# grep teardown_unused_network_gateway /etc/nova/nova.conf
#teardown_unused_network_gateway=false
teardown_unused_network_gateway=True

[apporc (appleorchard2000)]

I found this option is set just on primary controller, i don't know whether this is right. And i don't use nova-network, so can't test it.

[Bartłomiej Piotrowski (bpiotrowski)]

apporc: It is supposed to be false only on nova-network environments, so everything is fine IMHO.

[David A. Desrosiers (setuid)]

Years later, but adding a comment that this option doesn't appear to work in 100% of the cases, leaving empty bridges on remote hypervisors where no interfaces are attached.

Combined with another ebtables bug that appears to allow traffic to traverse the local bridge and br received by the remote, now empty bridge, it causes DNS and DHCP failures inside instances on the local bridge.

There appears to be a race condition between the time 'fip.host = None' is checked and when that API call commit the change to the db is flushed, where fip.host is checked too soon, and fails to remove the empty bridge.

The result, is an empty, remote bridge receiving traffic destined for a local bridge on another hypervisor.