server reboot hard and rebuild are flaky in tempest when ssh is enabled

[Andrea Frittoli]

Working on enabling back ssh access to VMs in tempest tests:

https://review.openstack.org/#/c/22415/
https://blueprints.launchpad.net/tempest/+spec/ssh-auth-strategy

On the gate devstack with nova networking the hard reboot and rebuild test are sometimes passing and sometimes not.

On the gate devstack with quantum networking the hard reboot and rebuild tests are systematically not passing, and blocking the overall blueprint implementation.

[Chuck Short (zulcss)]

can you provide examples please?

[Andrea Frittoli (andrea-frittoli)]

The current gate has "run_ssh" disabled.
I submitted a change to enable it, but different rechecks led to different results.
I'm trying to reproduce this on a local devstack to troubeshoot the issue, in the mean time I only have the logs from the failed gate runs:

[Andrea Frittoli (andrea-frittoli)]

http://logs.openstack.org/22415/14/check/gate-tempest-devstack-vm-postgres-full/5815
http://logs.openstack.org/22415/14/check/gate-tempest-devstack-vm-quantum/10888
http://logs.openstack.org/22415/11/check/gate-tempest-devstack-vm-full/6053

[Attila Fazekas (afazekas)]

It the VM actually survived the reboot and resize (looks like yes), probably https://review.openstack.org/#/c/23666/ fixes this too.

[Davanum Srinivas (DIMS) (dims-v)]

Andrea, will mark the nova bug as invalid. please reopen with New status if you have any new information

[Attila Fazekas (afazekas)]

Both issue still happening even with nova network when you try to access the machine via floating IP, after rebuild or reboot.

[Attila Fazekas (afazekas)]

A very basic shell script reproducer added.

[Adalberto Medeiros (adalbas)]

Moving to Confirmed in tempest as well, since this is still an issue there.

[Morgan Fainberg (mdrnstm)]

Based upon a conversation in IRC with Sean Dague, it appears this failure is related to this bug:

http://logs.openstack.org/97/62197/5/gate/gate-tempest-dsvm-full/9c67ae4/logs/screen-n-api.txt.gz#_2013-12-20_20_16_20_782

http://logs.openstack.org/97/62197/5/gate/gate-tempest-dsvm-full/9c67ae4/