Can't ping VM in overcloud

[Jan Provaznik]

CI job fails because a user instance created in overcloud is not pingable:

2014-08-06 08:29:01.174 | + wait_for 30 10 ping -c 1 192.0.2.46
2014-08-06 08:35:31.543 | Timing out after 300 seconds:
2014-08-06 08:35:31.543 | COMMAND=ping -c 1 192.0.2.46
2014-08-06 08:35:31.543 | OUTPUT=PING 192.0.2.46 (192.0.2.46) 56(84) bytes of data.
2014-08-06 08:35:31.543 | From 192.168.1.41 icmp_seq=1 Destination Host Unreachable

full log is here:
http://logs.openstack.org/96/105396/2/check-tripleo/check-tripleo-novabm-overcloud-f20-nonha/95d86f2/console.html

[Ben Nemec (bnemec)]

I've seen this on a number of CI runs today.

[Giulio Fidente (gfidente)]

nova-compute log pointing out an issue with the cinderclient when trying to boot the instance (from volume)

http://logs.openstack.org/51/112051/3/check-tripleo/check-tripleo-novabm-overcloud-f20-nonha/450e345/logs/overcloud-NovaCompute0_logs/nova-compute.txt.gz#_Aug_07_09_57_52

actual block device mapping passed to nova seems right indeed

http://logs.openstack.org/51/112051/3/check-tripleo/check-tripleo-novabm-overcloud-f20-nonha/450e345/console.html#_2014-08-07_10_02_11_299

[Giulio Fidente (gfidente)]

for reference, user image boot from volume was introduced by https://review.openstack.org/#/c/107437

[Ben Nemec (bnemec)]

I think the EndpointNotFound exception is probably the underlying problem here. The errors that happen after that are likely cascading failures from not being able to find Cinder.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112580
Committed: https://git.openstack.org/cgit/openstack/tripleo-incubator/commit/?id=f66ad982019e56e2c04b149ebadb8ccb7699dbd8
Submitter: Jenkins
Branch: master

commit f66ad982019e56e2c04b149ebadb8ccb7699dbd8
Author: Giulio Fidente <email address hidden>
Date: Thu Aug 7 15:43:33 2014 +0200

    Revert "devtest_overcloud: boot an instance from a volume"

    The change in subject seems to have brought up an issue
    breaking a good number of CI jobs, see bug 1353447.

    Until the root cause is found (and fixed), reverting this
    seems the only way to get CI back in a decent state.

    This reverts commit bc1da1fe067f237109f97c3894abc4296cc15a65.

    Change-Id: I10336b0c9f510ef87aa17fb5d53a5b5773b07177
    Closes-Bug: 1353447

[Dan Prince (dan-prince)]

I'm glad we have CI fixed. But the real issue here was caused by Nova breaking Cinder v1 API support for upgrades. This was a valid issue exposed by us simply having a boot from volume test in our CI (and why we should re-enable this test as soon as possible).

The Nova issue is here: https://bugs.launchpad.net/nova/+bug/1354499