quantum-adv-test fails in gate-tempest-devstack-vm-quantum

[Clark Boylan]

gate-tempest-devstack-vm-quantum fails running the quantum-adv-test devstack exercise. The relavent snip from a Jenkins console log is below and https://review.openstack.org/#/c/30938/1 is the failing change.

2013-06-03 23:01:28.772 | ++ confirm_server_active 9b1156c1-697a-4821-94fc-1e5292f0b981
2013-06-03 23:01:28.772 | ++ local VM_UUID=9b1156c1-697a-4821-94fc-1e5292f0b981
2013-06-03 23:01:28.772 | ++ timeout 90 sh -c 'while ! nova show 9b1156c1-697a-4821-94fc-1e5292f0b981 | grep status | grep -q ACTIVE; do sleep 1; done'
2013-06-03 23:02:58.740 | ++ echo 'server '\''9b1156c1-697a-4821-94fc-1e5292f0b981'\'' did not become active!'
2013-06-03 23:02:58.741 | server '9b1156c1-697a-4821-94fc-1e5292f0b981' did not become active!
2013-06-03 23:02:58.741 | ++ false
2013-06-03 23:02:58.741 | +++ failed
2013-06-03 23:02:58.741 | +++ local r=1
2013-06-03 23:02:58.741 | +++ set +o errtrace
2013-06-03 23:02:58.741 | +++ set +o xtrace
2013-06-03 23:02:58.741 | Failed to execute
2013-06-03 23:02:58.741 | Starting cleanup...
2013-06-03 23:03:04.447 | 2013-06-03 23:03:04.415 4853 WARNING quantum.debug.debug_agent [-] Failed to delete namespace qprobe-0e0b17cd-9708-4130-8c4a-6c53ba9f163f
2013-06-03 23:03:04.560 | Probe 0e0b17cd-9708-4130-8c4a-6c53ba9f163f deleted
2013-06-03 23:03:05.834 | Deleted network: 75ed8ad1-28f0-44e7-b97a-abf55f73b9b9
2013-06-03 23:03:09.602 | removed all tenants
2013-06-03 23:03:09.630 | Finished cleanup

[Hans Lindgren (hanlind)]

This is yet another case where a KeyError 'ports' exception is thrown by the compute node when it talks to quantum. See bug 1185834.

[clayg (clay-gerrard)]

saw the same test fail on https://review.openstack.org/#/c/32851/

found the KeyError in screen-n-cpu.txt.gz

    [req-c3f521f1-42c5-45f3-ac24-252319eb836a demo1 demo1] Exception during message handling
    Traceback (most recent call last):
      File "/opt/stack/new/nova/
        **args)
      File "/opt/stack/new/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
        result = getattr(proxyobj, method)(ctxt, **kwargs)
      File "/opt/stack/new/nova/nova/exception.py", line 98, in wrapped
        temp_level, payload)
      File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        self.gen.next()
      File "/opt/stack/new/nova/nova/exception.py", line 75, in wrapped
        return f(self, context, *args, **kw)
      File "/opt/stack/new/nova/nova/compute/manager.py", line 265, in decorated_function
        function(self, context, *args, **kwargs)
      File "/opt/stack/new/nova/nova/compute/manager.py", line 242, in decorated_function
        e, sys.exc_info())
      File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        self.gen.next()
      File "/opt/stack/new/nova/nova/compute/manager.py", line 229, in decorated_function
        return function(self, context, *args, **kwargs)
      File "/opt/stack/new/nova/nova/compute/manager.py", line 1507, in terminate_instance
        do_terminate_instance(instance, bdms)
      File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 246, in inner
        retval = f(*args, **kwargs)
      File "/opt/stack/new/nova/nova/compute/manager.py", line 1499, in do_terminate_instance
        reservations=reservations)
      File "/opt/stack/new/nova/nova/hooks.py", line 85, in inner
        rv = f(*args, **kwargs)
      File "/opt/stack/new/nova/nova/compute/manager.py", line 1453, in _delete_instance
        project_id=project_id)
      File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        self.gen.next()
      File "/opt/stack/new/nova/nova/compute/manager.py", line 1426, in _delete_instance
        self._shutdown_instance(context, instance, bdms)
      File "/opt/stack/new/nova/nova/compute/manager.py", line 1345, in _shutdown_instance
        network_info = self._get_instance_nw_info(context, instance)
      File "/opt/stack/new/nova/nova/compute/manager.py", line 761, in _get_instance_nw_info
        instance, conductor_api=self.conductor_api)
      File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 369, in get_instance_nw_info
        result = self._get_instance_nw_info(context, instance, networks)
      File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 377, in _get_instance_nw_info
        nw_info = self._build_network_info_model(context, instance, networks)
      File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 854, in _build_network_info_model
        network_IPs = self._nw_info_get_ips(client, port)
      File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 783, in _nw_info_get_ips
        client, fixed_ip['ip_address'], port['id'])
      File "/opt/stack/new/nova/nova/network/quantumv2/api.py", line 721, in _get_floating_ips_by_fixed_and_port
        port_id=port)
      Fil...