boot from volume + configdrive with virtio-scsi broken (regression)

[Mehdi Abaakouk]

Hi,

Since last ocata update (2:15.0.7-0ubuntu1~cloud0) "boot from volume"+configdrive is broken.

The libvirt xml generated looks wrong, on the first scsi disk the "unit" is wrong, it's 1 while it must be 0. The VM can't boot, kvm start but the boot screen show the "No boot disk found" message.

The wrong xml generated:

    <disk type='network' device='cdrom'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <auth username='openstack-service'>
        <secret type='ceph' uuid='XXXXXXXXXXXXXXXXX'/>
      </auth>
      <source protocol='rbd' name='disks/9029e91c-2c6e-4a83-b71f-4ee1055e51ca_disk.config'>
        <host name='XXX.XXX.XXX.XXX' port='6789'/>
        <host name='XXX.XXX.XXX.XXX' port='6789'/>
        <host name='XXX.XXX.XXX.XXX' port='6789'/>
      </source>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <auth username='openstack-service'>
        <secret type='ceph' uuid='XXXXXXXXXXXXXXXXX'/>
      </auth>
      <source protocol='rbd' name='ssds/volume-cb74e7c1-bcfc-4b52-b92a-f2750232734d'>
        <host name='XXX.XXX.XXX.XXX' port='6789'/>
        <host name='XXX.XXX.XXX.XXX' port='6789'/>
        <host name='XXX.XXX.XXX.XXX' port='6789'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <serial>cb74e7c1-bcfc-4b52-b92a-f2750232734d</serial>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

As workaround, I have fixed the issue here:

--- a/nova/virt/libvirt/driver.py 2016-03-21 00:08:47.702778684 +0100
+++ b/nova/virt/libvirt/driver.py 2016-03-21 00:21:19.877080690 +0100
@@ -3576,7 +3576,7 @@
                 LOG.debug('Config drive not found in RBD, falling back to the '
                           'instance directory', instance=instance)
         disk_info = disk_mapping[name]
- if 'unit' in disk_mapping:
+ if 'unit' in disk_mapping and disk_info["bus"] == "scsi":
             disk_unit = disk_mapping['unit']
             disk_mapping['unit'] += 1 # Increments for the next disk added
         conf = disk.libvirt_info(disk_info['bus'],

[Mehdi Abaakouk (sileht)]

The issue is still present on master branch.

[Logan V (loganv)]

The patch provided fixed the issue for me when doing virtio-scsi boot from RBD volume + config drive on Ocata.

However, I do see a lingering issue on an instance which has 2 virtio-scsi RBD volumes attached. One volume is the boot device and is marked bootable in Cinder. The other volume is a storage disk and is not marked bootable in cinder. Nova keeps generating the libvirt XML with the storage disk listed as LUN 0, and the boot disk listed as LUN 1, breaking the instance boot. Viewing the volume in Cinder shows the correct boot device listed as mounted at /dev/sda.

For reference below:
Boot volume: a63ea64e-a4fe-4bdb-98d9-a9a48e773f1a
Data volume: e82a7785-1c8c-4931-8948-4e16838f840e

ubuntu@7b25e53036d7:~$ openstack volume show e82a7785-1c8c-4931-8948-4e16838f840e
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'd83da61f-de88-4180-a434-71a64db3182e', u'attachment_id': u'c5f956ea-0c2d-43a2-8151-d481124b5a6e', u'attached_at': u'2017-11-20T12:07:22.000000', u'host_name': None, u'volume_id': u'e82a7785-1c8c-4931-8948-4e16838f840e', u'device': u'/dev/sdb', u'id': u'e82a7785-1c8c-4931-8948-4e16838f840e'}] |
| availability_zone | us-dfw-1 |
| bootable | false |
| consistencygroup_id | None ...

[Logan V (loganv)]

It seems like these 2 out of order virtio-scsi issues described above relating to secondary volumes and config drives ending up as index 0 are probably a regression introduced in https://review.openstack.org/#/q/topic:bug/1686116

This seriously impacts our environment due to heavy RBD use in nova/cinder. Most of our instances have at least some type of virtio-scsi backed disk. We've found that any instance using config drive or boot from volume + storage volume(s) is at risk of having an unbootable instance in Ocata or later when virtio-scsi is used.

[Logan V (loganv)]

Doing some more debugging, it looks like the disk_info['mapping'] is just processed in order:

disk_info={'disk_bus': u'scsi', 'cdrom_bus': 'ide', 'mapping': {u'/dev/sdb': {'bus': u'scsi', 'type': u'disk', 'dev': u'sdb'}, 'root': {'bus': u'scsi', 'boot_index': '1', 'type': u'disk', 'dev': u'sda'}, u'/dev/sda': {'bus': u'scsi', 'boot_index': '1', 'type': u'disk', 'dev': u'sda'}}}

I guess in _get_guest_storage_config this takes place, or maybe in _get_guest_disk_config, but since it seems to just iterate through the mapping in the order provided while incrementing the LUN each time, we end up with this situation.

[melanie witt (melwitt)]

Marking this as High because it's a regression.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/538310

[melanie witt (melwitt)]

Hi Logan,

Thank you for all the detailed info to help with debugging.

I noticed in your paste none of the volumes in your disk_info['mapping'] have boot_index=0. Is that correct? I thought one of them would have to have boot_index=0 if you have a bootable volume.

I'm trying to work out what's needed to address the remaining problem you're observing with boot-from-volume plus additional volumes and drives. Would it be possible for you to test the following patch that builds upon the first one?

--- a/nova/virt/libvirt/driver.py
+++ b/nova/virt/libvirt/driver.py
@@ -3763,6 +3763,8 @@ class LibvirtDriver(driver.ComputeDriver):
             # unit added and be able to increment it for each disk
             # added.
             disk_mapping['unit'] = 0
+ if self._is_booted_from_volume(block_device_info):
+ disk_mapping['unit'] = 1

         def _get_ephemeral_devices():
             eph_devices = []
@@ -3852,8 +3854,11 @@ class LibvirtDriver(driver.ComputeDriver):
             info = disk_mapping[vol_dev]
             self._connect_volume(context, connection_info, instance)
             if scsi_controller and scsi_controller.model == 'virtio-scsi':
- info['unit'] = disk_mapping['unit']
- disk_mapping['unit'] += 1
+ if vol.get('boot_index') == 0:
+ info['unit'] = 0
+ else:
+ info['unit'] = disk_mapping['unit']
+ disk_mapping['unit'] += 1
             cfg = self._get_volume_config(connection_info, info)
             devices.append(cfg)
             vol['connection_info'] = connection_info

[melanie witt (melwitt)]

Sorry that formatting came out all wrong. Here's a paste of it: https://pastebin.com/nDhjDcaw

[Logan V (loganv)]

Sorry I missed the patch you posted earlier until the bug update today.. I tested the patch and it fixes the situation in #2.

Confirmed @ the disk_info['mapping'] comment. None appear to have bood_index=0. We have quite a few boot from volume instances in the env, and only a few had this occur. The one in particular that I am testing with was created in Kilo timeframe. I wonder if something was not mapped quite right back then which has carried through the various upgrades. Anyway it seems like an edge case, and the patch does fix it. Thank you!

[melanie witt (melwitt)]

Thanks very much Logan for confirming that. We'll work to get the proposed patch [1] fixed up with proper unit test coverage and reviewed.

[1] https://review.openstack.org/538310

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/538310
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2616b384e642b6eb58eef7da87b6e893f25a949e
Submitter: Zuul
Branch: master

commit 2616b384e642b6eb58eef7da87b6e893f25a949e
Author: Jay Pipes <email address hidden>
Date: Fri Jan 26 12:20:35 2018 -0500

    only increment disk address unit for scsi devices

    We were erroneously incrementing the disk address unit attribute for
    non-scsi devices, which resulted in inconsistent disk device naming and
    addresses when SCSI devices were used along with non-SCSI devices (like
    configdrive devices).

    Also, we ensure that we assign unit number 0 for the boot volume of a
    boot-from-volume instance.

    Change-Id: Ia91e2f9c316e25394a0f41dc341d903dfcff6921
    Co-authored-by: Mehdi Abaakouk <email address hidden>
    Closes-bug: #1729584
    Closes-bug: #1753394

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/561196

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/561611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/561613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561196
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f9c66434eea245ae05a449059391515376f5a456
Submitter: Zuul
Branch: stable/queens

commit f9c66434eea245ae05a449059391515376f5a456
Author: Jay Pipes <email address hidden>
Date: Fri Jan 26 12:20:35 2018 -0500

    only increment disk address unit for scsi devices

    We were erroneously incrementing the disk address unit attribute for
    non-scsi devices, which resulted in inconsistent disk device naming and
    addresses when SCSI devices were used along with non-SCSI devices (like
    configdrive devices).

    Also, we ensure that we assign unit number 0 for the boot volume of a
    boot-from-volume instance.

    Change-Id: Ia91e2f9c316e25394a0f41dc341d903dfcff6921
    Co-authored-by: Mehdi Abaakouk <email address hidden>
    Closes-bug: #1729584
    Closes-bug: #1753394
    (cherry picked from commit 2616b384e642b6eb58eef7da87b6e893f25a949e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561611
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b255e16bd93d9891caff8ffc84b8d7bc2991f90a
Submitter: Zuul
Branch: stable/pike

commit b255e16bd93d9891caff8ffc84b8d7bc2991f90a
Author: Jay Pipes <email address hidden>
Date: Fri Jan 26 12:20:35 2018 -0500

    only increment disk address unit for scsi devices

    We were erroneously incrementing the disk address unit attribute for
    non-scsi devices, which resulted in inconsistent disk device naming and
    addresses when SCSI devices were used along with non-SCSI devices (like
    configdrive devices).

    Also, we ensure that we assign unit number 0 for the boot volume of a
    boot-from-volume instance.

    Co-authored-by: Mehdi Abaakouk <email address hidden>
    Closes-bug: #1729584
    Closes-bug: #1753394

     Conflicts:
     nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(artom) Conflicts in nova/tests/unit/virt/libvirt/test_driver.py
    because the surrounding _get_guest_config_with_graphics method isn't
    present in pike.

    Change-Id: Ia91e2f9c316e25394a0f41dc341d903dfcff6921
    (cherry picked from commit 2616b384e642b6eb58eef7da87b6e893f25a949e)
    (cherry picked from commit f9c66434eea245ae05a449059391515376f5a456)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561613
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1150d4a2af5b06c2df5dea053f5f7aea090c9145
Submitter: Zuul
Branch: stable/ocata

commit 1150d4a2af5b06c2df5dea053f5f7aea090c9145
Author: Jay Pipes <email address hidden>
Date: Fri Jan 26 12:20:35 2018 -0500

    only increment disk address unit for scsi devices

    We were erroneously incrementing the disk address unit attribute for
    non-scsi devices, which resulted in inconsistent disk device naming and
    addresses when SCSI devices were used along with non-SCSI devices (like
    configdrive devices).

    Also, we ensure that we assign unit number 0 for the boot volume of a
    boot-from-volume instance.

    Co-authored-by: Mehdi Abaakouk <email address hidden>
    Closes-bug: #1729584
    Closes-bug: #1753394

    Change-Id: Ia91e2f9c316e25394a0f41dc341d903dfcff6921
    (cherry picked from commit 2616b384e642b6eb58eef7da87b6e893f25a949e)
    (cherry picked from commit f9c66434eea245ae05a449059391515376f5a456)
    (cherry picked from commit b255e16bd93d9891caff8ffc84b8d7bc2991f90a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.1 release.