The setup:
- grizzly rc2 upgraded to grizzly (1:2013.1-0ubuntu1~cloud0), running on precise.
- fresh install configured based on what appears to be a folsom howto
Starting nova-network results in the following in the nova-network log:
...
2013-04-10 08:59:18.788 WARNING nova.db.sqlalchemy.api [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] Change will make usage less than 0 for the following resources: ['fixed_ips']
2013-04-10 08:59:19.098 ERROR nova.openstack.common.rpc.amqp [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] Exception during message handling
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 187, in deallocate_for_instance
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp super(FloatingIP, self).deallocate_for_instance(context, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 550, in deallocate_for_instance
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp self.deallocate_fixed_ip(context, fixed_ip['address'], host=host)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 243, in deallocate_fixed_ip
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp address)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 938, in deallocate_fixed_ip
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp self._teardown_network_on_host(context, network)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1630, in _teardown_network_on_host
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp self.driver.update_dhcp(elevated, dev, network)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 938, in update_dhcp
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp restart_dhcp(context, dev, network_ref)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1048, in restart_dhcp
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp _execute(*cmd, run_as_root=True)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1141, in _execute
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp return utils.execute(*cmd, **kwargs)
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp cmd=' '.join(cmd))
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Exit code: 3
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Stdout: ''
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp Stderr: 'sh: 1: /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: not found\n\ndnsmasq: cannot run lease-init script /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: No such file or directory\n'
2013-04-10 08:59:19.098 5003 TRACE nova.openstack.common.rpc.amqp
2013-04-10 08:59:19.103 ERROR nova.openstack.common.rpc.common [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] Returning exception Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro
Exit code: 3
Stdout: ''
Stderr: 'sh: 1: /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: not found\n\ndnsmasq: cannot run lease-init script /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: No such file or directory\n' to caller
2013-04-10 08:59:19.104 ERROR nova.openstack.common.rpc.common [req-8094076e-9813-42c4-968e-9dcb1f640659 69e6b81d4dbd41669313d522df780566 fdd2025e9254488dbe52919938c96edf] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data\n rval = self.proxy.dispatch(ctxt, version, method, **args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch\n return getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 187, in deallocate_for_instance\n super(FloatingIP, self).deallocate_for_instance(context, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 550, in deallocate_for_instance\n self.deallocate_fixed_ip(context, fixed_ip[\'address\'], host=host)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 243, in deallocate_fixed_ip\n address)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 938, in deallocate_fixed_ip\n self._teardown_network_on_host(context, network)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1630, in _teardown_network_on_host\n self.driver.update_dhcp(elevated, dev, network)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 938, in update_dhcp\n restart_dhcp(context, dev, network_ref)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner\n retval = f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1048, in restart_dhcp\n _execute(*cmd, run_as_root=True)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1141, in _execute\n return utils.execute(*cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute\n cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro\nExit code: 3\nStdout: \'\'\nStderr: \'sh: 1: /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: not found\\n\\ndnsmasq: cannot run lease-init script /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge: No such file or directory\\n\'\n']
2013-04-10 09:07:31.246 5003 INFO nova.service [-] Caught SIGTERM, exiting
2013-04-10 09:08:34.740 1334 INFO nova.manager [-] Skipping periodic task _periodic_update_dns because its interval is negative
2013-04-10 09:08:34.742 1334 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2013-04-10 09:08:34.761 1334 AUDIT nova.service [-] Starting network node (version 2013.1)
2013-04-10 09:09:04.146 1334 CRITICAL nova [-] Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-eth0.pid --listen-address=172.28.20.1 --except-interface=lo --dhcp-range=set:private,172.28.20.2,static,120s --dhcp-lease-max=1024 --dhcp-hostsfile=/var/lib/nova/networks/nova-eth0.conf --dhcp-script=/usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge --leasefile-ro
Exit code: 3
...
I initially created a symlink pointing /usr/lib/python2.7/dist-packages/bin/nova-dhcpbridge -> /usr/bin/nova-dhcpbridge
which fixed the problem, setting bindir=/usr/bin in nova.conf is a cleaner workaround.
I do not know if the default nova.conf has bindir specified, but it strikes me that /usr/bin is a much better default bindir than /var/lib/python2.7/dist-packages/bin...
lamont
This bug report relates to an unsupport Ubuntu release and an unsupported OpenStack release; in addition nova-network is officially deprecated in favour of neutron.
So marking this as a Won't Fix.