libvirt resize down prevention is invalid when using rbd as backend

Bug #1587802 reported by Yang Shengming
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Feodor Tersin

Bug Description

when using ceph as backend, instance can resize to a smaller flavor successfully without any exception, ResizeError failed to raise.

when using ceph as backend, xml file looks like this:
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='vms/e888e9c5-4c63-4ec3-a8fe-45a432cc82e0_disk'>
        <host name='172.16.40.18' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

As the disk type is network, so get_instance_disk_info cannot get disk_info
nova/virt/libvirt/driver.py:
    def _get_instance_disk_info(self, instance_name, xml,
                                block_device_info=None):
        ......
        for cnt, path_node in enumerate(path_nodes):
            disk_type = disk_nodes[cnt].get('type')
            path = path_node.get('file') or path_node.get('dev')
            target = target_nodes[cnt].attrib['dev']

            if not path:
                LOG.debug('skipping disk for %s as it does not have a path',
                          instance_name)
                continue

            if disk_type not in ['file', 'block']:
                LOG.debug('skipping disk because it looks like a volume', path)
                continue

            if target in volume_devices:
                LOG.debug('skipping disk %(path)s (%(target)s) as it is a '
                          'volume', {'path': path, 'target': target})
                continue
         ......

nova/virt/libvirt/driver.py:
    @staticmethod
    def _is_booted_from_volume(instance, disk_mapping):
        """Determines whether the VM is booting from volume

        Determines whether the disk mapping indicates that the VM
        is booting from a volume.
        """
        return ((not bool(instance.get('image_ref')))
                or 'disk' not in disk_mapping)

In the end when migrate_disk_and_power_off calling is_booted_from_volume, disk_mapping = [] and treated as volume backed instance.

Changed in nova:
assignee: nobody → Yang Shengming (yang-shengming)
Revision history for this message
Matt Riedemann (mriedem) wrote :

Which version of nova are you hitting this on? Does it exist in the latest release (mitaka)?

tags: added: ceph libvirt
Changed in nova:
status: New → Incomplete
Revision history for this message
Matt Riedemann (mriedem) wrote :

Also see:

https://ask.openstack.org/en/question/79054/openstack-kilo-cant-resize-instance-down/

Are you trying to resize down to a flavor that won't fit the image?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/325708

Changed in nova:
status: Incomplete → In Progress
Revision history for this message
Yang Shengming (yang-shengming) wrote :

Hi Matt, I'm exactly trying resize down to a flavor that won't fit the image. It comes to me on Liberty. After seeing the code on master, I guess it hasn't been fixed yet. I commit a patch which changed the is_booted_from_volume judgment.

Changed in nova:
assignee: Yang Shengming (yang-shengming) → Feodor Tersin (ftersin)
Revision history for this message
Feodor Tersin (ftersin) wrote :

Yang, sorry for changed assignee. It has been changed authomatically, and i can not set it back. Try yourself, pls.

Changed in nova:
assignee: Feodor Tersin (ftersin) → Yang Shengming (yang-shengming)
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
Timofey Durakov (tdurakov) wrote :

@yang-shengming hi, could you please attach traceback and libvirt's domain definition here.

description: updated
Changed in nova:
assignee: Yang Shengming (yang-shengming) → Feodor Tersin (ftersin)
Revision history for this message
melanie witt (melwitt) wrote :

I had meant to add some steps to reproduce the problem:

1. Set up devstack with ceph
2. Boot instance with a flavor with some disk
3. Resize down to a flavor with a smaller disk than step 2
4. Confirm resize
5. Shelve instance
6. Unshelve instance

Result: Instance will not be unshelved: “FlavorDiskSmallerThanImage: Flavor's disk is too small for requested image. Flavor disk is 1073741824 bytes, image is 5368709120 bytes.”

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Feodor Tersin (<email address hidden>) on branch: master
Review: https://review.openstack.org/334609
Reason: in favor of Ie424d172

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/382024
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ed994bb4d72b5f93cfb76a5f726264e4896ad4ed
Submitter: Jenkins
Branch: master

commit ed994bb4d72b5f93cfb76a5f726264e4896ad4ed
Author: Feodor Tersin <email address hidden>
Date: Tue Oct 4 20:17:17 2016 +0300

    libvirt: Improve _is_booted_from_volume implementation

    Currently this method cannot be used widely due to its parameters. It
    requires device_info - the dict which can not be easily obtained in many
    cases. Since it is often needed to figure out if an instance is booted
    from volume, and the method name is "appropriate", this sometimes leads
    to errors (when string result of get_instance_disk_info is passed as an
    argument to _is_booted_from_volume; see also Id5901254).

    This patch makes _is_booted_from_volume to use standard
    block_device_info structure, which is accessible in almost any driver
    method.

    Closes-bug: 1596957
    Closes-bug: 1587802
    Change-Id: Ie424d172ac9d6aeb42d83e512f2a18713134be3b

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b2

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/416558

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/325708
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the review queue sane. Please feel free to restore the change if you're still working on it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/416558
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c9ea2e179e7408f8b8f41579c6915be519c0b04e
Submitter: Jenkins
Branch: stable/newton

commit c9ea2e179e7408f8b8f41579c6915be519c0b04e
Author: Feodor Tersin <email address hidden>
Date: Tue Oct 4 20:17:17 2016 +0300

    libvirt: Improve _is_booted_from_volume implementation

    Currently this method cannot be used widely due to its parameters. It
    requires device_info - the dict which can not be easily obtained in many
    cases. Since it is often needed to figure out if an instance is booted
    from volume, and the method name is "appropriate", this sometimes leads
    to errors (when string result of get_instance_disk_info is passed as an
    argument to _is_booted_from_volume; see also Id5901254).

    This patch makes _is_booted_from_volume to use standard
    block_device_info structure, which is accessible in almost any driver
    method.

    Closes-bug: 1596957
    Closes-bug: 1587802
    Change-Id: Ie424d172ac9d6aeb42d83e512f2a18713134be3b
    (cherry picked from commit ed994bb4d72b5f93cfb76a5f726264e4896ad4ed)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.4

This issue was fixed in the openstack/nova 14.0.4 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.