check-tempest-dsvm-full-ceph failing on test_volume_boot_pattern scenario after 3/30

[Matt Riedemann]

https://review.openstack.org/#/c/165616/ enabled the test_volume_boot_pattern scenario test in tempest on 3/30 after this change to devstack https://review.openstack.org/#/c/165281/ for the underlying lvm lockup bug.

Once the test was enabled again, the check-tempest-dsvm-full-ceph job started failing hard again:

http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOlwiY2hlY2stdGVtcGVzdC1kc3ZtLWZ1bGwtY2VwaFwiIG1lc3NhZ2U6XCJ0ZXN0X3ZvbHVtZV9ib290X3BhdHRlcm5cIiBBTkQgbWVzc2FnZTpcIkZBSUxFRFwiIEFORCB0YWdzOlwiY29uc29sZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI3OTAxODkzODUyfQ==

There are errors in the n-cpu and g-api logs:

http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-ceph/07dcae0/logs/screen-n-cpu.txt.gz?level=TRACE

http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-ceph/07dcae0/logs/screen-g-api.txt.gz?level=TRACE

The n-cpu failures might be red herrings on a failed teardown of the instance b/c it didn't setup correctly given the g-api errors where it can't find the image.

Note that nova/cinder/glance are all configured in this job to use rbd/ceph backends.

[Matt Riedemann (mriedem)]

The image not found errors in g-api logs are not specific to the ceph job, so those are also red herrings:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVnaXN0cnkgY2xpZW50IHJlcXVlc3QgR0VUIC9pbWFnZXMvY2lycm9zLTAuMy4yLXg4Nl82NC11ZWMta2VybmVsIHJhaXNlZCBOb3RGb3VuZFwiIEFORCB0YWdzOlwic2NyZWVuLWctYXBpLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI3OTAyMTAxNTkzfQ==

[Matt Riedemann (mriedem)]

Bingo:

http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-ceph/07dcae0/logs/screen-n-cpu.txt.gz#_2015-04-01_14_34_35_696

2015-04-01 14:34:35.696 DEBUG nova.virt.libvirt.rbd_utils [req-c4beb2f7-21b9-4da3-bf7f-51e4a8955a25 None None] rbd image f72fc64f-6703-46c0-8687-810207085854_disk does not exist __init__ /opt/stack/new/nova/nova/virt/libvirt/rbd_utils.py:59

[Matt Riedemann (mriedem)]

Nevermind comment 2, another red herring, that is called from an existence check method.

[Matt Riedemann (mriedem)]

This should probably be duplicated against the cinder bug now.