instance can not resize ephemeral from mitaka to stein and master

Bug #1558880 reported by Tina Kevin on 2016-03-18
58
This bug affects 9 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Wishlist
Unassigned

Bug Description

Version
  mitaka

 Reproduce steps:
example:
* create a flavor with ephemeral disk
* boot an instance with the flavor
* resize the instance to a flavor with larger disk size and ephemeral disk size

Expected result:
* VM was running, disk and ephemeral are larger.

Actual result:
* VM was running, disk are larger but ephemeral are not larger.

This is because the VM ephemeral name is disk.eph0, but nova check is disk.local,
 this leads to ephemeral can not be extended.
I think it is unreasonable that ephemeral can not be extended.

Tina Kevin (song-ruixia) on 2016-03-18
tags: added: resize
tags: added: ephemeral
removed: resize
tags: added: resize
removed: ephemeral
Changed in nova:
assignee: nobody → SongRuixia (song-ruixia)
status: New → In Progress
Sylvain Bauza (sylvain-bauza) wrote :

Well, I meant possibly related to https://bugs.launchpad.net/nova/+bug/1558343 rather

Sylvain Bauza (sylvain-bauza) wrote :

Adding this one as a candidate for RC2 unless we verify it's not a regression.

tags: added: mitaka-rc-potential
Changed in nova:
status: In Progress → New
importance: Undecided → Medium
Matthew Booth (mbooth-9) wrote :

Haven't confirmed, but at first glance I don't see how this is obviously related to the config drive bug. I'll try to reproduce.

Matthew Booth (mbooth-9) wrote :

Can you post the exact boot command you used to create the instance? I'd like to know if the ephemeral was created implicitly or explicitly.

Matthew Booth (mbooth-9) wrote :

Nevermind, I can see that an implicitly-created ephemeral disk is created as eph0.

Matthew Booth (mbooth-9) wrote :

This has been the behaviour of Nova since at least change I8efd6af6706a097fb540e040a86ccbeaf131631f merged, which was September 2013. It may even be older than that, but the code before that was a bit hard to read. This isn't a recent regression, so certainly not an rc blocker. It's also not immediately obvious how to fix it robustly.

Sylvain Bauza (sylvain-bauza) wrote :

Given c#7, I think this is not a regression, so punting out of RC2

tags: removed: mitaka-rc-potential
Changed in nova:
status: New → Confirmed

Tina Kevin,
Are you working on the fix? Please change status to Inprogress if you are, otherwise change Assigned to ->nobody.

Tina Kevin (song-ruixia) on 2016-04-19
Changed in nova:
status: Confirmed → In Progress
tags: added: libvirt
Changed in nova:
importance: Medium → High

Here's a tempest test that demonstrates this bug:

    https://review.openstack.org/#/c/338411/

Tina Kevin (song-ruixia) on 2016-10-29
Changed in nova:
assignee: Tina Kevin (song-ruixia) → nobody
Sean Dague (sdague) on 2016-12-09
Changed in nova:
status: In Progress → Confirmed
importance: High → Medium

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/320759
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in nova:
assignee: nobody → Nazeema Begum (nazeema123)
Changed in nova:
assignee: Nazeema Begum (nazeema123) → nobody

The disk.local thing is a distraction, as that code is always a no-op in practice. The resize is done by the _create_image call.

For reference, the bug happened way before this in ComputeManager._finish_migration where we do:

        if old_instance_type_id != new_instance_type_id:
            instance_type = instance.get_flavor('new')
            self._set_instance_info(instance, instance_type)
            for key in ('root_gb', 'swap', 'ephemeral_gb'):
                if old_instance_type[key] != instance_type[key]:
                    resize_instance = True
                    break

The problem is that ephemeral disks are defined by BDMs, and not by instance.ephemeral_gb. The above code updates ephemeral_gb, but not the BDM. The libvirt driver is only looking in the BDM, so it doesn't see the resize.

Matthew Booth (mbooth-9) wrote :

The issue with automatically updating the BDM is that the BDM doesn't necessarily map directly to the flavor's ephemeral_gb.

By default, if a flavor has an ephemeral disk we will create a BDM of size flavor.ephemeral_gb for that disk when creating an instance. From this point on, the BDM is canonical.

However, a user may also specify BDMs explicitly at creation time, which means that the user may:

* Specify an ephemeral disk smaller than flavor.ephemeral_gb
* Specify multiple ephemeral disks whose total size is <= flavor.ephemeral_gb

In both of these cases, it is not at all clear what the behaviour on resize should be. In practise, the only safe thing would be to leave them alone. As there is no api to explicitly resize an ephemeral disk, and it's not clear that it's worth creating one, I don't see that there is any solution to the general case.

It is possible that we could check if there is a single, full size ephemeral disk and resize only in that case, possibly emitting a warning otherwise.

Sean Dague (sdague) wrote :

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/372721 in branch: master

Changed in nova:
status: Confirmed → In Progress
assignee: nobody → Nazeema Begum (nazeema123)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/372721
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Hello what is the status of this bug, is someone actively working on it ?
If not, releasing it by marking it confirmed or new would be good.

Bjoern Teipel (bjoern-teipel) wrote :

Removing assignee due to lack of progress/update. I'm running into this as well

Changed in nova:
assignee: Nazeema Begum (nazeema123) → nobody
status: In Progress → New
status: New → Confirmed
Steven Relf (srelf) wrote :

So i am also affected by this issue.

I was thinking. It sounds like we have the following items to work around.

1. ephemeral disk takes up the full allocation from flavor.
   if so upon resize we increase the current ephemeral disk to the new size

2. Multiple ephemeral disks, split into multiple disks upto the max allowed flavor size.
   In this case, could we not just create a new ephemeral disk that takes the full allocation of the additional ephemeral allocation from the new flavor.

3. Single or multi ephemeral disks which add up to less than the current flavor.
   This is the hardest one to handle i believe, i would be tempted to just build another single disk taking the full additional allocation. EG. first flavor has 200Gb Eph, but only 100Gb is used, then resized to a flavor with 400Gb eph, we keep the current 100Gb ephemeral, (as thats what they decided to use), then we would create a new eph disk out of the remainder so in this example 200Gb

leehom (feli5) wrote :

Hi, Is there any updates about this bug?

leehom (feli5) on 2019-04-23
summary: - instance can not resize ephemeral in mitaka
+ instance can not resize ephemeral from mitaka
summary: - instance can not resize ephemeral from mitaka
+ instance can not resize ephemeral from mitaka to stein
summary: - instance can not resize ephemeral from mitaka to stein
+ instance can not resize ephemeral from mitaka to stein and master
Tina Kevin (song-ruixia) wrote :

@leehom (feli5)
There should be no one

David Will (dingjian-sx) wrote :

@Tina Kevin
Hi! Isn't it recommended to adjust the ephemeral disk size?

Tina Kevin (song-ruixia) wrote :

Yes

leehom (feli5) wrote :

Or we can provide API to support add new ephemeral disks.
As it is supported to create multiple ephemeral disks as long as the total size not greater than the size defined in flavor.
In case there are multiple ephemeral disks. It's hard to determine which disk should be extended.

Matt Riedemann (mriedem) wrote :

This is so latent and complicated to resolve that I've marked it as wishlist since it would probably require a spec to describe all of the detailed issues involved in fixing this and what the options and alternatives would be.

Changed in nova:
importance: Medium → Wishlist

Reviewed: https://review.opendev.org/691915
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d3e38143a7e46d06a6632b41b5484b13e81884cf
Submitter: Zuul
Branch: master

commit d3e38143a7e46d06a6632b41b5484b13e81884cf
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 29 12:50:13 2019 -0400

    Add known limitation about resize not resizing ephemeral disks

    It's a long known issue that resize does not resize ephemeral
    disks attached to the server yet we continue to get bugs about
    it and there is no one working on fixing it. As such this change
    adds a note to the API reference for the resize server action API
    mentioning the known limitation.

    Change-Id: I31ca3627f2d3ef5a8db5eecf3759cd304b059963
    Related-Bug: #1558880

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers