instance can not resize ephemeral in mitaka

Bug #1558880 reported by Tina Kevin on 2016-03-18
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Nazeema Begum

Bug Description

Version
  mitaka

 Reproduce steps:
example:
* create a flavor with ephemeral disk
* boot an instance with the flavor
* resize the instance to a flavor with larger disk size and ephemeral disk size

Expected result:
* VM was running, disk and ephemeral are larger.

Actual result:
* VM was running, disk are larger but ephemeral are not larger.

This is because the VM ephemeral name is disk.eph0, but nova check is disk.local,
 this leads to ephemeral can not be extended.
I think it is unreasonable that ephemeral can not be extended.

Tina Kevin (song-ruixia) on 2016-03-18
tags: added: resize
tags: added: ephemeral
removed: resize
tags: added: resize
removed: ephemeral
Changed in nova:
assignee: nobody → SongRuixia (song-ruixia)
status: New → In Progress
Sylvain Bauza (sylvain-bauza) wrote :

Well, I meant possibly related to https://bugs.launchpad.net/nova/+bug/1558343 rather

Sylvain Bauza (sylvain-bauza) wrote :

Adding this one as a candidate for RC2 unless we verify it's not a regression.

tags: added: mitaka-rc-potential
Changed in nova:
status: In Progress → New
importance: Undecided → Medium
Matthew Booth (mbooth-9) wrote :

Haven't confirmed, but at first glance I don't see how this is obviously related to the config drive bug. I'll try to reproduce.

Matthew Booth (mbooth-9) wrote :

Can you post the exact boot command you used to create the instance? I'd like to know if the ephemeral was created implicitly or explicitly.

Matthew Booth (mbooth-9) wrote :

Nevermind, I can see that an implicitly-created ephemeral disk is created as eph0.

Matthew Booth (mbooth-9) wrote :

This has been the behaviour of Nova since at least change I8efd6af6706a097fb540e040a86ccbeaf131631f merged, which was September 2013. It may even be older than that, but the code before that was a bit hard to read. This isn't a recent regression, so certainly not an rc blocker. It's also not immediately obvious how to fix it robustly.

Sylvain Bauza (sylvain-bauza) wrote :

Given c#7, I think this is not a regression, so punting out of RC2

tags: removed: mitaka-rc-potential
Changed in nova:
status: New → Confirmed

Tina Kevin,
Are you working on the fix? Please change status to Inprogress if you are, otherwise change Assigned to ->nobody.

Tina Kevin (song-ruixia) on 2016-04-19
Changed in nova:
status: Confirmed → In Progress
tags: added: libvirt
Changed in nova:
importance: Medium → High
Diana Clarke (diana-clarke) wrote :

Here's a tempest test that demonstrates this bug:

    https://review.openstack.org/#/c/338411/

Tina Kevin (song-ruixia) on 2016-10-29
Changed in nova:
assignee: Tina Kevin (song-ruixia) → nobody
Sean Dague (sdague) on 2016-12-09
Changed in nova:
status: In Progress → Confirmed
importance: High → Medium

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/320759
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in nova:
assignee: nobody → Nazeema Begum (nazeema123)
Changed in nova:
assignee: Nazeema Begum (nazeema123) → nobody
Matthew Booth (mbooth-9) wrote :

The disk.local thing is a distraction, as that code is always a no-op in practice. The resize is done by the _create_image call.

For reference, the bug happened way before this in ComputeManager._finish_migration where we do:

        if old_instance_type_id != new_instance_type_id:
            instance_type = instance.get_flavor('new')
            self._set_instance_info(instance, instance_type)
            for key in ('root_gb', 'swap', 'ephemeral_gb'):
                if old_instance_type[key] != instance_type[key]:
                    resize_instance = True
                    break

The problem is that ephemeral disks are defined by BDMs, and not by instance.ephemeral_gb. The above code updates ephemeral_gb, but not the BDM. The libvirt driver is only looking in the BDM, so it doesn't see the resize.

Matthew Booth (mbooth-9) wrote :

The issue with automatically updating the BDM is that the BDM doesn't necessarily map directly to the flavor's ephemeral_gb.

By default, if a flavor has an ephemeral disk we will create a BDM of size flavor.ephemeral_gb for that disk when creating an instance. From this point on, the BDM is canonical.

However, a user may also specify BDMs explicitly at creation time, which means that the user may:

* Specify an ephemeral disk smaller than flavor.ephemeral_gb
* Specify multiple ephemeral disks whose total size is <= flavor.ephemeral_gb

In both of these cases, it is not at all clear what the behaviour on resize should be. In practise, the only safe thing would be to leave them alone. As there is no api to explicitly resize an ephemeral disk, and it's not clear that it's worth creating one, I don't see that there is any solution to the general case.

It is possible that we could check if there is a single, full size ephemeral disk and resize only in that case, possibly emitting a warning otherwise.

Sean Dague (sdague) wrote :

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/372721 in branch: master

Changed in nova:
status: Confirmed → In Progress
assignee: nobody → Nazeema Begum (nazeema123)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers