Server fails to become active because: Failed to add interface: Dump was interrupted and may be inconsistent

[Sean Dague]

Since the trusty upgrade of nodes we're seeing a new timing issue where computes don't build due to a race around network interface adding. The n-cpu traceback that we see in these cases is:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/new/nova/nova/compute/manager.py", line 1544, in _allocate_network_async
    dhcp_options=dhcp_options)
  File "/opt/stack/new/nova/nova/network/api.py", line 46, in wrapped
    return func(self, context, *args, **kwargs)
  File "/opt/stack/new/nova/nova/network/base_api.py", line 61, in wrapper
    res = f(self, context, *args, **kwargs)
  File "/opt/stack/new/nova/nova/network/api.py", line 253, in allocate_for_instance
    nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
  File "/opt/stack/new/nova/nova/network/rpcapi.py", line 175, in allocate_for_instance
    macs=jsonutils.to_primitive(macs))
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
    retry=self.retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
    timeout=timeout, retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
    retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
    raise result
NovaException_Remote: Failed to add interface: Dump was interrupted and may be inconsistent.

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/new/nova/nova/network/floating_ips.py", line 118, in allocate_for_instance
    **kwargs)

  File "/opt/stack/new/nova/nova/network/manager.py", line 525, in allocate_for_instance
    requested_networks=requested_networks)

  File "/opt/stack/new/nova/nova/network/manager.py", line 213, in _allocate_fixed_ips
    vpn=vpn, address=address)

  File "/opt/stack/new/nova/nova/network/manager.py", line 945, in allocate_fixed_ip
    exc_info=True)

  File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/opt/stack/new/nova/nova/network/manager.py", line 927, in allocate_fixed_ip
    self._setup_network_on_host(context, network)

  File "/opt/stack/new/nova/nova/network/manager.py", line 1746, in _setup_network_on_host
    self.l3driver.initialize_gateway(network)

  File "/opt/stack/new/nova/nova/network/l3.py", line 104, in initialize_gateway
    gateway=(network_ref['gateway'] is not None))

  File "/opt/stack/new/nova/nova/network/linux_net.py", line 1380, in plug
    return _get_interface_driver().plug(network, mac_address, gateway)

  File "/opt/stack/new/nova/nova/network/linux_net.py", line 1428, in plug
    network, gateway)

  File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 249, in inner
    return f(*args, **kwargs)

  File "/opt/stack/new/nova/nova/network/linux_net.py", line 1571, in ensure_bridge
    raise exception.NovaException(msg)

NovaException: Failed to add interface: Dump was interrupted and may be inconsistent.

Removing descriptor: 3

An example failure is here - http://logs.openstack.org/23/95723/8/check/check-tempest-dsvm-postgres-full/e1494ef/logs/screen-n-cpu.txt.gz?level=TRACE

[Dean Troyer (dtroyer)]

I'm not certain this is a complete dupe, I have a very similar trace that appears to still occur occasionally:

http://logs.openstack.org/56/113856/7/check/check-grenade-dsvm-partial-ncpu/d71c938/logs/old/screen-n-cpu.txt.gz

2014-08-19 20:18:27.082 31657 ERROR oslo.messaging._drivers.common [-] Returning exception Failed to add interface: Dump was interrupted and may be inconsistent.

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/old/nova/nova/network/manager.py", line 1314, in setup_networks_on_host
    call_func(context, network)

  File "/opt/stack/old/nova/nova/network/manager.py", line 1660, in _setup_network_on_host
    self.l3driver.initialize_gateway(network)

  File "/opt/stack/old/nova/nova/network/l3.py", line 104, in initialize_gateway
    gateway=(network_ref['gateway'] is not None))

  File "/opt/stack/old/nova/nova/network/linux_net.py", line 1384, in plug
    return _get_interface_driver().plug(network, mac_address, gateway)

  File "/opt/stack/old/nova/nova/network/linux_net.py", line 1432, in plug
    network, gateway)

  File "/opt/stack/old/nova/nova/openstack/common/lockutils.py", line 249, in inner
    return f(*args, **kwargs)

  File "/opt/stack/old/nova/nova/network/linux_net.py", line 1575, in ensure_bridge
    raise exception.NovaException(msg)

NovaException: Failed to add interface: Dump was interrupted and may be inconsistent.