boot_index ignorance when booting VM

[Volodymyr Litovka]

Dear colleagues,

seems I'm experiencing the issue, similar to https://bugs.launchpad.net/nova/+bug/1570107 but on Queens. When I try to boot VM with two volumes attached (both volumes made from bootable images and are bootable as well) and use boot_index, Nova ignores boot_index regardless of whether I use or don't use disk_bus/device_type. I see the following debug in nova-api.log:

"block_device_mapping_v2": [
{"source_type": "volume", "delete_on_termination": false, "boot_index": 0, "uuid": "df338f65-71b2-4735-909b-29693f8f80c8", "destination_type": "volume"},
{"source_type": "volume", "delete_on_termination": false, "boot_index": -1, "uuid": "3461d1ec-102a-4372-a009-2d693a93a884", "destination_type": "volume"}]

but VM booted from uuid "3461d" (both are of the same bus_type - first one is sda, 2nd is sdb). Absolutely same result for the another configuration:

"block_device_mapping_v2": [
{"boot_index": 0, "uuid": "08c94175-0491-4339-ac39-b5a5aab51037", "disk_bus": "virtio", "source_type": "volume", "device_type": "disk", "destination_type": "volume", "delete_on_termination": false},
{"boot_index": -1, "uuid": "39184a9b-32cf-49fe-b3f9-5a98fdec9c24", "disk_bus": "virtio", "source_type": "volume", "device_type": "disk", "destination_type": "volume", "delete_on_termination": false}]

VM booted from "39184" (which is vdb, marked with -1), while another volume (08c94, vda) marked with "0".

And even more - if I change order in configuration in the way:

"block_device_mapping_v2": [
{"boot_index": -1, "uuid": "1a6672b7-c196-4e91-b157-92432741d6d7", "disk_bus": "virtio", "source_type": "volume", "device_type": "disk", "destination_type": "volume", "delete_on_termination": false},
{"boot_index": 0, "uuid": "eee78267-385a-4bae-b53f-8ee4ec2e64e9", "disk_bus": "virtio", "source_type": "volume", "device_type": "disk", "destination_type": "volume", "delete_on_termination": false}]

I get booted from "-1" volume. Well, may be, some info in images metadata influence boot order? These are:

- Image which always boot in all three examples above:
hw_disk_bus='scsi', hw_qemu_guest_agent='yes', hw_scsi_model='virtio-scsi', img_hide_hypervisor_id='true', locations='[{u'url': u'cinder://0fb40899-d22e-465d-981b-ce35d07180c3', u'metadata': {}}]

- Image, which I want to boot from:
hw_boot_menu='true', hw_disk_bus='scsi', hw_qemu_guest_agent='yes', hw_scsi_model='virtio-scsi', img_hide_hypervisor_id='true', locations='[{u'url': u'swift+config://ref1/glance/7a71b12c-4a1e-479f-8676-cc2e671b7cc4', u'metadata': {}}]

If this matters - Swift is CEPH Object Storage (for second image).

And surprise - SOMETIMES (in rare cases) VM boots with correct image, but I can't find any system in this behavior.

- My environment is:
Operating system: Ubuntu 16.04
Openstack: Queens (Nova 2:17.0.5-0ubuntu1~cloud0, Glance 2:16.0.1-0ubuntu1.1~cloud0)

ADDITIONAL INFO:

- For the last example:
#virsh domblklist instance-0000077d
Target Source
------------------------------------------------
vda volumes/volume-eee78267-385a-4bae-b53f-8ee4ec2e64e9
vdb volumes/volume-1a6672b7-c196-4e91-b157-92432741d6d7

and autogenerated libvirt config for the domain is available here - https://pastebin.com/N51Kzysb

I use Heat to produce these configurations and config is the following:

  n1:
    type: OS::Nova::Server
    properties:
      flavor: ...
      config_drive: False
      name: jex-n1
      block_device_mapping_v2:
        - volume_id: { get_resource: n1-attach }
          delete_on_termination: false
          device_type: disk
          disk_bus: virtio
          boot_index: -1
        - volume_id: { get_resource: n1-vol }
          delete_on_termination: false
          device_type: disk
          disk_bus: virtio
          boot_index: 0

  n1-vol:
    type: OS::Cinder::Volume
    properties:
      size: 8
      name: jex-n1-vol
      image: bionic-Qpub

  n1-attach:
    type: OS::Cinder::Volume
    properties:
      size: 8
      name: jex-n1-att
      image: xenial

I will highly appreciate if anybody can help to solve this issue. Any additional information can be provided by request.

Thank you.

[Volodymyr Litovka (doka.ua)]

The issue is inside VM, not Nova's one.