Expensive Device allocation loop when NBD is absent

[Davanum Srinivas (DIMS)]

See log stash query:
"Device allocation failed. Will retry in 2 seconds."

Example:
http://logs.openstack.org/10/163910/6/check/gate-rally-dsvm-cinder/46d89ab/logs/screen-n-cpu.txt.gz#_2015-03-30_14_55_56_941

Logs:
2015-03-30 14:55:56.940 ERROR nova.virt.disk.mount.nbd [req-df13155c-0340-4e10-aec4-2fabea382eef None None] nbd module not loaded
2015-03-30 14:55:56.940 DEBUG oslo_concurrency.lockutils [req-df13155c-0340-4e10-aec4-2fabea382eef None None] Lock "nbd-allocation-lock" released by "_inner_get_dev" :: held 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-03-30 14:55:56.941 INFO nova.virt.disk.mount.api [req-df13155c-0340-4e10-aec4-2fabea382eef None None] Device allocation failed. Will retry in 2 seconds.
2015-03-30 14:55:58.941 DEBUG oslo_concurrency.lockutils [req-df13155c-0340-4e10-aec4-2fabea382eef None None] Lock "nbd-allocation-lock" acquired by "_inner_get_dev" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
2015-03-30 14:55:58.941 ERROR nova.virt.disk.mount.nbd [req-df13155c-0340-4e10-aec4-2fabea382eef None None] nbd module not loaded

We should not even enter the loop when /sys/block/nbd0 is clearly absent.

[Davanum Srinivas (DIMS) (dims-v)]

Probably the cause of the Rally slowness found here:
http://logs.openstack.org/36/168636/3/check/gate-rally-dsvm-rally/ce18c9f/rally-plot/results.html.gz#/NovaServers.boot_and_delete_server

[Davanum Srinivas (DIMS) (dims-v)]

Compare to an old run:
http://logs.openstack.org/42/160142/8/check/gate-rally-dsvm-rally/76fecde/rally-plot/results.html.gz#/NovaServers.boot_and_delete_server

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/168950

[Sean Dague (sdague)]

This only exposes in the rally and cells jobs, I wonder if there is just a broken configuration here - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIm5iZCBtb2R1bGUgbm90IGxvYWRlZFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI3NzM1NDA2Njg4LCJtb2RlIjoic2NvcmUiLCJhbmFseXplX2ZpZWxkIjoiYnVpbGRfbmFtZSJ9

It's also quite new

[Sean Dague (sdague)]

Nova is being asked to do a thing, which it can't, which is put files in the guest. This retry is actually kind of important.

The bug really is exposing the fact that *no* working file injection mechanism exists in these rally jobs, but they are asking guests to do something.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: master
Review: https://review.openstack.org/168950