Launch instance failed - Error Failed to add interface: sudo: cannot get working directory

[David Naori]

Launch instance failed - Error Failed to add interface: sudo: cannot get working directory

2012-06-04 12:35:47 ERROR nova.compute.manager [req-34ca021e-3245-4308-a5c5-60cc463bf2f4 2f4edfa99cab42de92eacda360043116 7eed65dd55474b9e94cd412d9f66b406] [instance: 13a852d8-e047-4289-a19f-1651221d9122] Instance failed network setup
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] Traceback (most recent call last):
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 559, in _allocate_network
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] requested_networks=requested_networks)
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 170, in allocate_for_instance
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] 'args': args})
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] File "/usr/lib/python2.6/site-packages/nova/rpc/__init__.py", line 68, in call
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] return _get_impl().call(context, topic, msg, timeout)
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 521, in call
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 338, in call
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] rv = list(rv)
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 306, in __iter__
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] raise result
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] RemoteError: Remote error: Error Failed to add interface: sudo: cannot get working directory
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122]
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122] [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 253, in _process_data\n rval
= node_func(context=ctxt, **node_args)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 257, in wrapped\n return func(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/net
work/manager.py", line 320, in allocate_for_instance\n **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 257, in wrapped\n return func(self, context, *args, **kwargs)\n', u' File "/usr/lib/pyt
hon2.6/site-packages/nova/network/manager.py", line 906, in allocate_for_instance\n requested_networks=requested_networks)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 213, in _allocate_fixed_ips\n
vpn=vpn, address=address)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1197, in allocate_fixed_ip\n self._setup_network_on_host(context, network)\n', u' File "/usr/lib/python2.6/site-packages/nova/net
work/manager.py", line 1709, in _setup_network_on_host\n self.l3driver.initialize_gateway(network)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/l3.py", line 98, in initialize_gateway\n gateway=(network_ref[\'gateway\
'] is not None))\n', u' File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 895, in plug\n return _get_interface_driver().plug(network, mac_address, gateway)\n', u' File "/usr/lib/python2.6/site-packages/nova/net
work/linux_net.py", line 940, in plug\n network, gateway)\n', u' File "/usr/lib/python2.6/site-packages/nova/utils.py", line 944, in inner\n retval = f(*args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/nova/network/l
inux_net.py", line 1042, in ensure_bridge\n raise exception.Error(\'Failed to add interface: %s\' % err)\n', u'Error: Failed to add interface: sudo: cannot get working directory\n\n'].
2012-06-04 12:35:47 TRACE nova.compute.manager [instance: 13a852d8-e047-4289-a19f-1651221d9122]
2012-06-04 12:35:47 ERROR nova.rpc.amqp [req-34ca021e-3245-4308-a5c5-60cc463bf2f4 2f4edfa99cab42de92eacda360043116 7eed65dd55474b9e94cd412d9f66b406] Exception during message handling
2012-06-04 12:35:47 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-06-04 12:35:47 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-06-04 12:35:47 TRACE nova.rpc.amqp return f(*args, **kw)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 177, in decorated_function
2012-06-04 12:35:47 TRACE nova.rpc.amqp sys.exc_info())
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-06-04 12:35:47 TRACE nova.rpc.amqp self.gen.next()
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 171, in decorated_function
2012-06-04 12:35:47 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 651, in run_instance
2012-06-04 12:35:47 TRACE nova.rpc.amqp do_run_instance()
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 946, in inner
2012-06-04 12:35:47 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 650, in do_run_instance
2012-06-04 12:35:47 TRACE nova.rpc.amqp self._run_instance(context, instance_uuid, **kwargs)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 451, in _run_instance
2012-06-04 12:35:47 TRACE nova.rpc.amqp self._set_instance_error_state(context, instance_uuid)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-06-04 12:35:47 TRACE nova.rpc.amqp self.gen.next()
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 424, in _run_instance
2012-06-04 12:35:47 TRACE nova.rpc.amqp requested_networks)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 559, in _allocate_network
2012-06-04 12:35:47 TRACE nova.rpc.amqp requested_networks=requested_networks)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 170, in allocate_for_instance
2012-06-04 12:35:47 TRACE nova.rpc.amqp 'args': args})
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/__init__.py", line 68, in call
2012-06-04 12:35:47 TRACE nova.rpc.amqp return _get_impl().call(context, topic, msg, timeout)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py", line 521, in call
2012-06-04 12:35:47 TRACE nova.rpc.amqp return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 338, in call
2012-06-04 12:35:47 TRACE nova.rpc.amqp rv = list(rv)
2012-06-04 12:35:47 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 306, in __iter__
2012-06-04 12:35:47 TRACE nova.rpc.amqp raise result
2012-06-04 12:35:47 TRACE nova.rpc.amqp RemoteError: Remote error: Error Failed to add interface: sudo: cannot get working directory
2012-06-04 12:35:47 TRACE nova.rpc.amqp

* Tried to restart nova-network service - it got stuck upon startup

 2012-06-05 11:18:04 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=8202) inner /usr/lib/python2.6/site-packages/nova/utils.py:928
2012-06-05 11:18:04 DEBUG nova.utils [-] Got semaphore "iptables" for method "apply"... from (pid=8202) inner /usr/lib/python2.6/site-packages/nova/utils.py:932
2012-06-05 11:18:04 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=8202) inner /usr/lib/python2.6/site-packages/nova/utils.py:936

* removed the lock file
#rm -f /var/lib/nova/tmp/nova-iptables.lock

restarted the service and everything worked out...

[Mark McLoughlin (markmc)]

Ok, the error was in ensure_bridge()

  Failed to add interface: sudo: cannot get working directory

The code in question is:

            out, err = _execute('ip', 'addr', 'show', 'dev', interface,
                                'scope', 'global', run_as_root=True)
            ....
            if (err and err != "device %s is already a member of a bridge;"
                     "can't enslave it to bridge %s.\n" % (interface, bridge)):
                raise exception.Error('Failed to add interface: %s' % err)

Seems fairly bizarre that we'd get this error

Google doesn't show up anyone else seeing this problem

The bug wasn't reproducible and I know the reporter is no longer working on OpenStack

Moving to Invalid