Attempt to boot from volume - no image supplied

[Derek Higgins]

About 12 UTC 29/1/2015 all tripleo overcloud jobs started failing with

2015-01-29 17:32:21.475 | + wait_for -w 50 --delay 5 -- neutron port-list -f csv -c id --quote none '|' grep id
2015-01-29 17:33:11.364 | Timing out after 50 seconds:
2015-01-29 17:33:11.364 | COMMAND=neutron port-list -f csv -c id --quote none | grep id
2015-01-29 17:33:11.364 | OUTPUT=
2015-01-29 17:33:11.364 | + cleanup

reproducing this localy shows a failure in getting the block device
| created | 2015-01-30T02:43:05Z
                                                                                            |
| fault | {"message": "Build of instance 4c958085-7a95-4825-af04-cf574c3614c7 aborted: Failure prepping block device.", "code": 500, "details": " File \"/opt/stack/venvs/nova/lib/
python2.7/site-packages/nova/compute/manager.py\", line 2077, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/manager.py\", line 2192, in _build_and_run_instance |
| | 'create.error', fault=e) |

[Derek Higgins (derekh)]

A revert of this nova commit seems to fix the problem
https://review.openstack.org/#/c/143054/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/151616

[Daniel Berrange (berrange)]

That tiny snippet of log messages isn't really very helpful in seeing what's wrong. Can you provide the full nova compute log file at least

[Derek Higgins (derekh)]

An example of the error can be found here
http://logs.openstack.org/06/150206/6/check-tripleo/check-tripleo-ironic-overcloud-f20-nonha/3b1268c/console.html

The controler logs are here
http://logs.openstack.org/06/150206/6/check-tripleo/check-tripleo-ironic-overcloud-f20-nonha/3b1268c/logs/ov-vq6pik5r3zr-0-6hyrfeafla5u-Controller_logs/

[Matt Riedemann (mriedem)]

http://logs.openstack.org/06/150206/6/check-tripleo/check-tripleo-ironic-overcloud-f20-nonha/3b1268c/logs/ov-i6a3r2vbqm-1-qrud6cls6p6r-NovaCompute_logs/nova-compute.txt.gz#_Jan_30_09_29_00

Jan 30 09:29:00 ov-i6a3r2vbqm-1-qrud6cls6p6r-novacompute-mpl6wabwvv2p nova-compute[4285]: 2015-01-30 09:29:00,119 ERROR Failure prepping block device
                                                                                          Traceback (most recent call last):
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2274, in _build_resources
                                                                                              block_device_mapping)
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 1811, in _default_block_device_names
                                                                                              root_bdm)
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 1770, in _default_root_device_name
                                                                                              root_bdm)
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6186, in default_root_device_name
                                                                                              cdrom_bus)
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 432, in get_root_info
                                                                                              root_bdm, disk_bus)
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 375, in get_info_from_bdm
                                                                                              device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 189, in find_disk_dev_for_disk_bus
                                                                                              if not has_disk_dev(mapping, disk_dev):
                                                                                            File "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 105, in has_disk_dev
                                                                                              info = mapping[disk]
                                                                                   ...

[Matt Riedemann (mriedem)]

So it looks like something is passing the mapping as a list rather than a dict, but tracing the code calls back in that stack trace I'm not sure where that is happening.

[Daniel Berrange (berrange)]

Base on the logs at http://logs.openstack.org/94/151594/1/check-tripleo/check-tripleo-ironic-overcloud-precise-nonha/2c71080/logs/ this looks like the relevant error message:

Jan 30 11:29:46 overcloud-novacompute1-54zbppby55dk nova-compute: 2015-01-30 11:29:46,190 ERROR Failure prepping block device#012Traceback (most recent call last):#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2274, in _build_resources#012 block_device_mapping)#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1811, in _default_block_device_names#012 root_bdm)#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1770, in _default_root_device_name#012 root_bdm)#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6186, in default_root_device_name#012 cdrom_bus)#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 432, in get_root_info#012 root_bdm, disk_bus)#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 375, in get_info_from_bdm#012 device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 189, in find_disk_dev_for_disk_bus#012 if not has_disk_dev(mapping, disk_dev):#012 File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 105, in has_disk_dev#012 info = mapping[disk]#012TypeError: string indices must be integers, not str

Need to figure out what's causing that type error exception - should be fairly painless to solve i imagine

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/151658

[Daniel Berrange (berrange)]

It is also curious that Tempest didn't catch the broken boot from volume in Nova integration tests. Probably a sign of a test case that could be added

[OpenStack Infra (hudson-openstack)]

Change abandoned by Derek Higgins (<email address hidden>) on branch: master
Review: https://review.openstack.org/151616
Reason: Tested the patch from dan locally on tripleo and seems to be ok
    https://review.openstack.org/151658

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151658
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0ba91324362edb9d1339cb41c145d1aa90f9fd5f
Submitter: Jenkins
Branch: master

commit 0ba91324362edb9d1339cb41c145d1aa90f9fd5f
Author: Daniel P. Berrange <email address hidden>
Date: Fri Jan 30 14:55:50 2015 +0000

    libvirt: fix missing block device mapping parameter

    This change caused a regression:

      commit 024ac74ffbc47239426fa95580851825d67dbe61
      Author: Daniel P. Berrange <email address hidden>
      Date: Tue Dec 16 14:24:23 2014 +0000

        libvirt: always pass image meta when getting disk info from bdm

    A merge rebase mistake there caused the block device mapping
    parameter to get dropped when when get_root_info calls
    get_info_from_bdm. For added fun the test case had the same
    mistake so everything looked fine. The result was that boot from
    volume was broken

    Closes-bug: #1416321
    Change-Id: Ieaf2d6c456f8bcc4bc7a0ffe7dc912c7cc8b3dc6