block device mapping : booting up attached volumes with no boot index when multiple bootable volumes are used

[ESC Dev]

Description
===========

provide 2 block device mappings, both bootable volumes with only 1 of them having a boot_index (i.e. only 1 specified as boot device) and both have different bus types. Observe booted instance is actually an instance of the image with no boot_index!

Steps to reproduce
==================

Created volume from cirros image
Created volume from ubuntu image

Attempted to create instance sourced from ubuntu volume and also have the cirros volume attached
- set boot index to 0 for ubuntu volume
- set no boot index for cirros (in trial 2, i set it to -1 explicitly so it would only be an attached volume)
- bus type provided for ubuntu volume was 'ide'
- bus type provided for cirros volume was 'virtio'
- send server payload
- observed the instance is booted from the cirros volume despite having no boot index!

sample payload :

{"server":{"flavorRef":"8e41c8ce-8bca-4012-92f0-6d83dc4e7dfe","name":"vol-test_cirros-bootable_0_cb3378f2-01d1-4662-8c9e-df6cff05c056","networks":[{"port":"1c11a3f9-47e6-4374-9890-54b5dd91f7c2"}],"block_device_mapping_v2":[{"uuid":"03b3c938-ed05-41a3-af92-5344431892ae","source_type":"volume","destination_type":"volume","delete_on_termination":"false","disk_bus": "virtio" },{"uuid":"7cebbbc0-113e-452b-b577-9043140656bf","boot_index":"0","source_type":"volume","destination_type":"volume","delete_on_termination":"false","disk_bus": "ide" },{"uuid":"5870a3d6-f475-4312-b26f-5162d76b9074","source_type":"volume","destination_type":"volume","delete_on_termination":"false","disk_bus": "virtio" }]}

NOTE : when the ubuntu volume is the only volume and the bus type is IDE, the created instance is an ubuntu instance
Also, if both volumes have the bus types set to virtio, the ubuntu instance is correctly booted.
Appears to be some bus type related issue, but it wasn't documented anywhere and there's no error.

Expected result
===============

Boot from the volume with boot index provided or fail with error

Actual result
=============

Boots from the other volume, even though it has no boot index.

Environment
===========
Openstack Kilo

hypervisor : libvirt + KVM
storage : LVM
networking : neutron with openvswitch

[zhengyue (zhengyue-5)]

I think this because the boot sequence in libvirt: http://libvirt.org/formatdomain.html#elementsOSBIOS.

In the section boot:
"Thus, e.g., a domain configured to boot from "hd" with vdb, hda, vda, and hdc disks assigned to it will boot from vda (the sorted list is vda, vdb, hda, hdc). Similar domain with hdc, vda, vdb, and hda disks will boot from hda (sorted disks are: hda, hdc, vda, vdb)."

The block_device_mapping_v2 in your sample:
[{"uuid":"03b3c938-ed05-41a3-af92-5344431892ae","source_type":"volume","destination_type":"volume","delete_on_termination":"false","disk_bus": "virtio" },{"uuid":"7cebbbc0-113e-452b-b577-9043140656bf","boot_index":"0","source_type":"volume","destination_type":"volume","delete_on_termination":"false","disk_bus": "ide" },{"uuid":"5870a3d6-f475-4312-b26f-5162d76b9074","source_type":"volume","destination_type":"volume","delete_on_termination":"false","disk_bus": "virtio" }]

Before instance create the device_name will set for each device:
[{
    "uuid":"03b3c938-ed05-41a3-af92-5344431892ae",
    "source_type":"volume",
    "device_name":"/dev/vda", # Add device_name by disk_bus, the virtio mapping to "vd*"
    "destination_type":"volume",
    "delete_on_termination":"false",
    "disk_bus": "virtio" },
{
    "uuid":"7cebbbc0-113e-452b-b577-9043140656bf",
    "boot_index":"0",
    "source_type":"volume",
    "device_name": "/dev/hda", # ide mapping to "hd*"
    "destination_type":"volume",
    "delete_on_termination":"false",
    "disk_bus": "ide" },
{
    "uuid":"5870a3d6-f475-4312-b26f-5162d76b9074",
    "source_type":"volume",
    "device_name":"/dev/vdb",
    "destination_type":"volume",
    "delete_on_termination":"false",
    "disk_bus": "virtio" }]

I think the disk sequence after sorted by libvirt is:
1. /dev/vda
2. /dev/vdb
3. /dev/hda

[ESC Dev (oadeboye)]

Thanks for the information 'Zhengyue'. I think you are missing the comment on the expected behavior. Basically i have specified only 1 boot index, in my alternate tests i explicitly set the boot index of the other volumes to (-1) to ensure they are not booted from according to the block device mapping documentation : http://docs.openstack.org/developer/nova/block_device_mapping.html

 "Some hypervisors may not support booting from multiple devices, so will only consider the device with boot index of 0. Some hypervisors will support booting from multiple devices, but only if they are of different types - eg a disk and CD-ROM. Setting a negative value or None indicates that the device should not be used for booting. The simplest usage is to set it to 0 for the boot device and leave it as None for any other devices."

Given the above excerpt, the wrong instance shouldnt be booted at all.

[Nodir Kodirov (knodir)]

I am not able to reproduce the bug on devstack (running inside VM). The behaviour I see is correct as per documentation (http://docs.openstack.org/developer/nova/block_device_mapping.html). Note that I am using QEMU as a backend (not KVM). See details below.

I created two volumes:
1) bootable Cirros (volume ID=45d6a4e4-c478-4b28-9850-049b5c9a8d51)
2) bootable Ubuntu (volume ID=2bffb48d-c859-4df1-a2a7-66e98af43837)

When both volumes are "available", I created VM by specifying Ubuntu as a bootable (bootindex=0) and Cirros as (bootindex=-1)
$ nova --debug boot --flavor 2 --block-device source=volume,id=2bffb48d-c859-4df1-a2a7-66e98af43837,dest=volume,size=5,shutdown=preserve,bootindex=0 --block-device source=volume,id=45d6a4e4-c478-4b28-9850-049b5c9a8d51,dest=volume,size=1,shutdown=preserve,bootindex=-1 uvm
[...]
DEBUG (session:248) REQ: curl -g -i -X POST http://192.168.229.137:8774/v2.1/os-volumes_boot -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}db2d307d5a2df66c6cb0a53c3cd7b0d8d77314e5" -d '{"server": {"name": "uvm", "imageRef": "", "block_device_mapping_v2": [{"boot_index": "0", "uuid": "2bffb48d-c859-4df1-a2a7-66e98af43837", "volume_size": "5", "source_type": "volume", "destination_type": "volume", "delete_on_termination": false}, {"boot_index": "-1", "uuid": "45d6a4e4-c478-4b28-9850-049b5c9a8d51", "volume_size": "1", "source_type": "volume", "destination_type": "volume", "delete_on_termination": false}], "flavorRef": "2", "max_count": 1, "min_count": 1}}'
[...]

VM successfully created with Ubuntu image (as expected).

I repeated the same experiment by swapping bootable index values, i.e., Ubuntu bootindex=-1 and Cirros bootindex=0.
$ nova --debug boot --flavor 2 --block-device source=volume,id=2bffb48d-c859-4df1-a2a7-66e98af43837,dest=volume,size=5,shutdown=preserve,bootindex=-1 --block-device source=volume,id=45d6a4e4-c478-4b28-9850-049b5c9a8d51,dest=volume,size=1,shutdown=preserve,bootindex=0 cvm
[...]
DEBUG (session:248) REQ: curl -g -i -X POST http://192.168.229.137:8774/v2.1/os-volumes_boot -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}12de1ea7163bebdf6a77566061f24036a69941dd" -d '{"server": {"name": "cvm", "imageRef": "", "block_device_mapping_v2": [{"boot_index": "-1", "uuid": "2bffb48d-c859-4df1-a2a7-66e98af43837", "volume_size": "5", "source_type": "volume", "destination_type": "volume", "delete_on_termination": false}, {"boot_index": "0", "uuid": "45d6a4e4-c478-4b28-9850-049b5c9a8d51", "volume_size": "1", "source_type": "volume", "destination_type": "volume", "delete_on_termination": false}], "flavorRef": "2", "max_count": 1, "min_count": 1}}'

VM successfully created with Cirros image (as expected).

[ESC Dev (oadeboye)]

Thanks for confirming it is a disk_bus related issue then Nodir.
Your payload left out the suspected cause of the problem "disk_bus" .
At least this confirms the boot_index may not be respected if disk_bus attribute is provided like in my case.

[Augustina Ragwitz (auggy)]

Added "needs-attention" tag because this hasn't been touched since April. It looks like the key issue that needs to be reproduced here is that boot_index is ignored when disk_bus is provided.

[Sean Dague (sdague)]

Automatically discovered version kilo in description. If this is incorrect, please update the description to include 'nova version: ...'