test_create_image_with_reboot fails with UnexpectedTaskStateError in gate-nova-python*

[Matt Riedemann]

Similar to bug 1266611 but with a different failure:

http://logs.openstack.org/63/71063/15/gate/gate-nova-python27/9ecf426/console.html.gz

http://logs.openstack.org/47/69047/15/check/gate-nova-python26/38dcf86/console.html#_2014-03-09_23_37_24_801

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVW5leHBlY3RlZFRhc2tTdGF0ZUVycm9yOiBVbmV4cGVjdGVkIHRhc2sgc3RhdGU6IGV4cGVjdGluZyBbTm9uZV0gYnV0IHRoZSBhY3R1YWwgc3RhdGUgaXMgcG93ZXJpbmctb2ZmXCIgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIgQU5EIChidWlsZF9uYW1lOlwiZ2F0ZS1ub3ZhLXB5dGhvbjI3XCIgT1IgYnVpbGRfbmFtZTpcImdhdGUtbm92YS1weXRob24yNlwiKSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDY1OTA0MTkwMX0=

4 hits in 7 days, and this shows up on the unclassified e-r bugs page.

[Matt Riedemann (mriedem)]

e-r query patch: https://review.openstack.org/80091

[Ken'ichi Ohmichi (oomichi)]

I investigated this problem according to http://logs.openstack.org/85/105485/4/gate/gate-nova-python26/3c56855/ , and the cause seems the same as https://bugs.launchpad.net/nova/+bug/1311778 .
That would be race between periodic tasks and REST API task.

WARNING [nova.compute.manager] Instance is not stopped. Calling the stop API.
ERROR [nova.compute.manager] error during stop() in sync_power_state.
Traceback (most recent call last):
  File "nova/compute/manager.py", line 5557, in _sync_instance_power_state
    self.compute_api.force_stop(context, db_instance)
  File "nova/compute/api.py", line 1767, in force_stop
    self.compute_rpcapi.stop_instance(context, instance, do_cast=do_cast)
  File "nova/compute/rpcapi.py", line 908, in stop_instance
    return rpc_method(ctxt, 'stop_instance', instance=instance)
  File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call
    wait_for_reply=True, timeout=timeout)
  File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send
    timeout=timeout)
  File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_fake.py", line 166, in send
    return self._send(target, ctxt, message, wait_for_reply, timeout)
  File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_fake.py", line 161, in _send
    'No reply on topic %s' % target.topic)
MessagingTimeout: No reply on topic compute
WARNING [nova.service] Service killed that has no database entry
}}}

Traceback (most recent call last):
  File "nova/tests/api/ec2/test_cloud.py", line 2532, in test_create_image_with_reboot
    self._do_test_create_image(False)
  File "nova/tests/api/ec2/test_cloud.py", line 2505, in _do_test_create_image
    no_reboot=no_reboot)
  File "nova/api/ec2/cloud.py", line 1752, in create_image
    self.compute_api.start(context, instance)
  File "nova/compute/api.py", line 182, in inner
    return function(self, context, instance, *args, **kwargs)
  File "nova/compute/api.py", line 173, in wrapped
    return function(self, context, instance, *args, **kwargs)
  File "nova/compute/api.py", line 209, in _wrapped
    return fn(self, context, instance, *args, **kwargs)
  File "nova/compute/api.py", line 163, in inner
    return f(self, context, instance, *args, **kw)
  File "nova/compute/api.py", line 1786, in start
    instance.save(expected_task_state=[None])
  File "nova/objects/base.py", line 196, in wrapper
    return fn(self, ctxt, *args, **kwargs)
  File "nova/objects/instance.py", line 469, in save
    columns_to_join=_expected_cols(expected_attrs))
  File "nova/db/api.py", line 780, in instance_update_and_get_original
    columns_to_join=columns_to_join)
  File "nova/db/sqlalchemy/api.py", line 167, in wrapper
    return f(*args, **kwargs)
  File "nova/db/sqlalchemy/api.py", line 2238, in instance_update_and_get_original
    columns_to_join=columns_to_join)
  File "nova/db/sqlalchemy/api.py", line 2289, in _instance_update
    actual=actual_state, expected=expected)...

[Joe Gordon (jogo)]

No hits in the past 10 days looks like this bug was resolved or it now has a different fingerprint