FAIL: tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest.test_run_stop_terminate_instance -- nova VM fails to go to running state when using neutron

[Davanum Srinivas (DIMS)]

http://logs.openstack.org/10/49210/2/check/check-tempest-devstack-vm-neutron/1a9306e/logs/screen-n-cpu.txt.gz

2013-10-01 23:57:11.785 2937 INFO nova.compute.manager [-] [instance: 771f75a5-a649-4946-b495-a57558915b03] During sync_power_state the instance has a pending task. Skip.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/queue.py", line 107, in switch
    self.greenlet.switch(value)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 447, in _dispatch_thread
    self._dispatch_events()
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 507, in _dispatch_events
    assert _c
AssertionError

http://logs.openstack.org/10/49210/2/check/check-tempest-devstack-vm-neutron/1a9306e/testr_results.html.gz

ft71.3: tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest.test_run_stop_terminate_instance[gate,smoke]_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{2013-10-01 23:57:07,853 state: pending}}}

Traceback (most recent call last):
  File "tempest/thirdparty/boto/test_ec2_instance_run.py", line 150, in test_run_stop_terminate_instance
    self.assertInstanceStateWait(instance, "running")
  File "tempest/thirdparty/boto/test.py", line 356, in assertInstanceStateWait
    state = self.waitInstanceState(lfunction, wait_for)
  File "tempest/thirdparty/boto/test.py", line 341, in waitInstanceState
    self.valid_instance_state)
  File "tempest/thirdparty/boto/test.py", line 331, in state_wait_gone
    state = state_wait(lfunction, final_set, valid_set)
  File "tempest/thirdparty/boto/utils/wait.py", line 57, in state_wait
    (dtime, final_set, status))
AssertionError: State change timeout exceeded!(400s) While waitingfor set(['running', '_GONE']) at "pending"

More importantly, this failure does NOT have "Cannot understand JSON" and is thus not the same as bug : https://bugs.launchpad.net/tempest/+bug/1223161

[Joe Gordon (jogo)]

Looks like this is a nova + neutron bug only:

http://logstash.openstack.org/#eyJzZWFyY2giOiJcIlN0YXRlIGNoYW5nZSB0aW1lb3V0IGV4Y2VlZGVkISg0MDBzKSBXaGlsZSB3YWl0aW5nZm9yIHNldChbJ3J1bm5pbmcnLCAnX0dPTkUnXSkgYXQgXFxcInBlbmRpbmdcXFwiXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDcyNjk4NDg4OCwibW9kZSI6InRlcm1zIiwiYW5hbHl6ZV9maWVsZCI6IkBmaWVsZHMuYnVpbGRfbmFtZSJ9

[Joe Gordon (jogo)]

I dug into this yesterday, and it appears that nova-compute is missing its 'Got sempahore / lockl 'update_usage" call so the DB doesn't know the instance is booted. Because this only happens with neutron there is probably some strange interaction, such as a synchronous call to neutron that hangs for some reason.

http://logs.openstack.org/98/49198/1/check/check-tempest-devstack-vm-neutron/e3cd6d8/logs/screen-n-cpu.txt.gz#_2013-10-01_16_57_22_043

[Joe Gordon (jogo)]

18:08 < jog0> http://logs.openstack.org/67/49367/1/check/check-tempest-devstack-vm-neutron/f333c6f/logs/screen-q-agt.txt.gz?
18:08 < jog0> search for '4a5a7f6e78ab' that is the failed instance-id
18:08 < jog0> I See it 843 times!
18:08 < jog0> is that right?
18:10 < markmcclain> yeah.. that is the result of polling the vswitch
18:10 < jog0> for a passing copy of q-agt I only see 'vm-uuid' under 300 times
18:11 < jog0> it looks like the vswitch is polled until the end of time

[Russell Bryant (russellb)]

The logstash query is no longer finding any hits on this one.

[Cedric Brandily (cbrandily)]

This bug is > 365 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]