resize of instance, that was booted from volume, trigger image size check then fails resize

[Wendy Mitchell]

Brief Description
-----------------
resize of instance, that was booted from volume, trigger image size check then fails resize

Severity
--------
standard

Steps to Reproduce
------------------

1. Create two flavors based on originating flavor and destination flavor

| 15566866-06d7-487a-b087-9fe928ec8c00 | flavor | 1024 | 4 | 0 | | 1 | 1.0 | True
 flavor:extra_specs | {"stx_storage": "remote", "hw:mem_page_size": "2048"}
flavor:ram | 1024
|
| flavor:swap | 0
|
| flavor:vcpus | 1
| flavor:disk | 4
|
| flavor:ephemeral | 0

| 20ebe3fe-c1a3-4e62-8c34-10c435f27112 | flavor-2 | 1024 | 1 | 1 | 512 | 1 | 1.0 | True
| extra_specs | {"stx_storage": "remote", "hw:mem_page_size": "2048"}
name | flavor-2
| ram | 1024
| swap | 512
| vcpus | 1
| disk | 1

2. Boot instance from volume with an originating flavor
OS-EXT-SRV-ATTR:hostname | tenant2-vm-13
id | 04f43c9e-d3fa-46c4-9605-c0d1cf8dda71
os-extended-volumes:volumes_attached | [{"id": "672aaa60-1086-45ce-8650-8195aa8dd0da", "delete_on_termination": false}
OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
OS-EXT-SRV-ATTR:instance_name | instance-000000a4

3. Resize instance to destination flavor that has smaller disk size. Confirm resize.

[2019-04-14 10:05:04,590]
nova resize --poll 04f43c9e-d3fa-46c4-9605-c0d1cf8dda71 20ebe3fe-c1a3-4e62-8c34-10c435f27112'

Expected Behavior
------------------
Did not expect resize of instance, that was booted from volume, to trigger image size check

Actual Behavior
----------------
[2019-04-14 10:05:06,427]
Resize of instance booted from volume inadvertently triggered the image size check and results in a ClientExpection related to FlavorDiskSmalerThanMinDisk
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.FlavorDiskSmallerThanMinDisk'> (HTTP 500

nfv-vim.log (controller-0)

2019-04-14T10:05:06.228 controller-0 VIM_Thread[244649] INFO _instance_director.py.151 Instance action allowed for tenant2-vm-13, action_type=resize

2019-04-14T10:05:06.228 controller-0 VIM_Thread[244649] DEBUG _instance.py.1968 Resize instance tenant2-vm-13.

2019-04-14T10:05:06.228 controller-0 VIM_Thread[244649] INFO _instance_state_initial.py.36 Exiting state (initial) for tenant2-vm-13.

2019-04-14T10:05:06.229 controller-0 VIM_Thread[244649] INFO _instance_state_resize.py.27 Entering state (resize) for tenant2-vm-13.
2019-04-14T10:05:06.229 controller-0
VIM_Thread[244649] DEBUG nfvi_compute_api.py.3418 Instance action rest-api post path: /v2.1/6b4999fd6e414ae1b3eace662eb2ed84/servers/04f43c9e-d3fa-46c4-9605-c0d1cf8dda71/action.
2019-04-14T10:05:06.270 controller-0
VIM_Event-Log_Thread[244975] INFO fm.py.379 Generated customer log, fm_uuid=621d5dc6-79a8-40a7-814f-2809cbadb859.

2019-04-14T10:05:06.375 controller-0 VIM_Thread[244649] ERROR Caught exception while trying to resize an instance, error=[OpenStack Rest-API Exception: method=POST, url=http://nova-api.openstack.svc.cluster.local:8774/v2.1/6b4999fd6e414ae1b3eace662eb2ed84/servers/04f43c9e-d3fa-46c4-9605-c0d1cf8dda71/action, headers={'wrs-header': 'true', 'Content-Type': 'application/json', 'X-OpenStack-Nova-API-Version': '2.72'}, body={"resize": {"flavorRef": "20ebe3fe-c1a3-4e62-8c34-10c435f27112"}}, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Content-Length', '217'), ('X-Compute-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6'), ('Vary', 'OpenStack-API-Version, X-OpenStack-Nova-API-Version'), ('Openstack-Api-Version', 'compute 2.72'), ('X-Openstack-Nova-Api-Version', '2.72'), ('Date', 'Sun, 14 Apr 2019 10:05:06 GMT'), ('Content-Type', 'application/json; charset=UTF-8'), ('X-Openstack-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6')], response_body={"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.FlavorDiskSmallerThanMinDisk'>", "code": 500}}].

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_compute_api.py", line 2127, in resize_instance
    future.result = (yield)
OpenStackRestAPIException: [OpenStack Rest-API Exception: method=POST, url=http://nova-api.openstack.svc.cluster.local:8774/v2.1/6b4999fd6e414ae1b3eace662eb2ed84/servers/04f43c9e-d3fa-46c4-9605-c0d1cf8dda71/action, headers={'wrs-header': 'true', 'Content-Type': 'application/json', 'X-OpenStack-Nova-API-Version': '2.72'}, body={"resize": {"flavorRef": "20ebe3fe-c1a3-4e62-8c34-10c435f27112"}}, status_code=500, reason=HTTP Error 500: Internal Server Error, response_headers=[('Content-Length', '217'), ('X-Compute-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6'), ('Vary', 'OpenStack-API-Version, X-OpenStack-Nova-API-Version'), ('Openstack-Api-Version', 'compute 2.72'), ('X-Openstack-Nova-Api-Version', '2.72'),
('Date', 'Sun, 14 Apr 2019 10:05:06 GMT'), ('Content-Type', 'application/json; charset=UTF-8'), ('X-Openstack-Request-Id', 'req-aca6c954-1171-48ce-9286-3e9b952481d6')], response_body={"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.FlavorDiskSmallerThanMinDisk'>", "code": 500}}]

2019-04-14T10:05:06.377 controller-0 VIM_Thread[244649] INFO nfvi_compute_api.py.918 Sent response for request 8b44ee79-9371-479b-83a4-50534ddef1c7.
2019-04-14T10:05:06.377 controller-0
VIM_Thread[244649]
DEBUG _instance_task_work.py.426 Resize-Instance callback for tenant2-vm-13, response={'completed': False, 'reason': "unexpected api error. please report this at http://bugs.launchpad.net/nova/ and attach the nova api log if possible.\n<class 'nova.exception.flavordisksmallerthanmindisk'>"}.
2019-04-14T10:05:06.378 controller-0 VIM_Thread[244649] DEBUG _instance_tasks.py.367 Task (resize-instance_tenant2-vm-13) complete.

2019-04-14T10:05:06.378 controller-0 VIM_Thread[244649] INFO _instance_state_resize.py.68 Resize failed for tenant2-vm-13.
2019-04-14T10:05:06.378 controller-0
 VIM_Thread[244649]
DEBUG _instance.py.2179 Resize failed for instance tenant2-vm-13, reason=unexpected api error. please report this at http://bugs.launchpad.net/nova/ and attach the nova api log if possible.
<class 'nova.exception.flavordisksmallerthanmindisk'>.

Reproducibility
---------------
yes

System Configuration
--------------------
storage
(Lab: IP_20_27 20190414 10:02:43 test_resize_vm_positive[remote-4_0_0-1_1_512-volume])

Branch/Pull Time/Commit
--------------------
Load: 20190410T013000Z
Job: STX_build_master_master

[Matt Riedemann (mriedem)]

Are you sure this is reproducible in pure nova code? Because this looks like forked nova code for starlingx:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_compute_api.py", line 2127, in resize_instance
    future.result = (yield)

Looking at this code in the compute API in pure nova, it looks like that error only gets raised here:

https://github.com/openstack/nova/blob/e25d59078e61fe9f925dbef53dfe88e575d34dab/nova/compute/api.py#L655

Which is being called from resize now:

https://github.com/openstack/nova/blob/e25d59078e61fe9f925dbef53dfe88e575d34dab/nova/compute/api.py#L3652

As a result of this change in stein:

https://review.openstack.org/#/c/620706/

Note that root_bdm=None there so that's how we get into the else block of _validate_flavor_image_nostatus because it's not considering the instance as volume-backed.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/653268

[Matt Riedemann (mriedem)]

Hmm, even by passing the root_bdm to _validate_flavor_image_nostatus it can fail with VolumeSmallerThanMinDisk if the volume_size is smaller than the min_disk size of the backing image.

I think the original intent behind passing root_bdm=None to _validate_flavor_image_nostatus during resize was to just side-step these disk size checks for volume-backed servers but that doesn't seem to be easy without passing some parameter to _validate_flavor_image_nostatus to just tell it to ignore size checks.

[Matt Riedemann (mriedem)]

My fix is hitting a side effect of bug 1646740 where the min_disk stored in the image metadata for the server is based on the flavor root_gb rather than the volume size...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/653273

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653268
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a40c7f3e8d06f0bc8918a25b60f73ec2c18add05
Submitter: Zuul
Branch: master

commit a40c7f3e8d06f0bc8918a25b60f73ec2c18add05
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:00:15 2019 -0400

    Add functional regression recreate test for bug 1825020

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein
    started validating extra specs on the new flavor during a
    resize but didn't take into account some validation that
    happens with disk size with the new flavor if the validation
    code isn't told that the server is volume-backed. That results
    in a 500 FlavorDiskSmallerThanMinDisk response from the API
    if the new flavor has a smaller disk size than the current
    flavor that the server is using, which shouldn't actually
    matter for a volume-backed server.

    This adds a functional regression test to recreate the bug.

    Change-Id: I9e7e44727946705506d740896349ca1833cfddcd
    Related-Bug: #1825020

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/656511

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653273
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8aae3e39e58a05ed50b891aeae261641f544ad1f
Submitter: Zuul
Branch: master

commit 8aae3e39e58a05ed50b891aeae261641f544ad1f
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:53:03 2019 -0400

    Fix volume-backed resize with a smaller disk flavor

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein started
    calling _validate_flavor_image_nostatus during resize but did not
    pass the root_bdm since the underlying volume does not change. This
    can, however, lead to FlavorDiskSmallerThanMinDisk being raised
    if the new flavor has a disk size smaller than the current flavor.
    This is wrong in the case of a volume-backed server because we don't
    care about the root disk size in the flavor in that case.

    This fixes the bug by splitting the pci/numa validation logic out
    of _validate_flavor_image_nostatus into a separate method so that
    the resize method can call that directly for a volume-backed server
    rather than deal with the complicated disk size logic in
    _validate_flavor_image_nostatus (see bug 1646740 for details, but
    tl;dr if the image min_disk is less than the flavor root_gb, the
    min_disk stored in the instance image system metadata is the
    flavor['root_gb'] which could be larger than the root volume size).
    To see why trying to use _validate_flavor_image_nostatus during
    resize for a volume-backed server is a mess, look at PS1 of this
    change.

    Change-Id: I91c9c1e88afa035c757f692c78ad72d69cc3c431
    Closes-Bug: #1825020

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/656666

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/656511
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a1db4507ac832913a911afbee90f76f576d8ea20
Submitter: Zuul
Branch: stable/stein

commit a1db4507ac832913a911afbee90f76f576d8ea20
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:00:15 2019 -0400

    Add functional regression recreate test for bug 1825020

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein
    started validating extra specs on the new flavor during a
    resize but didn't take into account some validation that
    happens with disk size with the new flavor if the validation
    code isn't told that the server is volume-backed. That results
    in a 500 FlavorDiskSmallerThanMinDisk response from the API
    if the new flavor has a smaller disk size than the current
    flavor that the server is using, which shouldn't actually
    matter for a volume-backed server.

    This adds a functional regression test to recreate the bug.

    Change-Id: I9e7e44727946705506d740896349ca1833cfddcd
    Related-Bug: #1825020
    (cherry picked from commit a40c7f3e8d06f0bc8918a25b60f73ec2c18add05)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/656666
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3285210add2758ad0182358317fabca626cdb0ee
Submitter: Zuul
Branch: stable/stein

commit 3285210add2758ad0182358317fabca626cdb0ee
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 16 23:53:03 2019 -0400

    Fix volume-backed resize with a smaller disk flavor

    Change I06fad233006c7bab14749a51ffa226c3801f951b in Stein started
    calling _validate_flavor_image_nostatus during resize but did not
    pass the root_bdm since the underlying volume does not change. This
    can, however, lead to FlavorDiskSmallerThanMinDisk being raised
    if the new flavor has a disk size smaller than the current flavor.
    This is wrong in the case of a volume-backed server because we don't
    care about the root disk size in the flavor in that case.

    This fixes the bug by splitting the pci/numa validation logic out
    of _validate_flavor_image_nostatus into a separate method so that
    the resize method can call that directly for a volume-backed server
    rather than deal with the complicated disk size logic in
    _validate_flavor_image_nostatus (see bug 1646740 for details, but
    tl;dr if the image min_disk is less than the flavor root_gb, the
    min_disk stored in the instance image system metadata is the
    flavor['root_gb'] which could be larger than the root volume size).
    To see why trying to use _validate_flavor_image_nostatus during
    resize for a volume-backed server is a mess, look at PS1 of this
    change.

    Change-Id: I91c9c1e88afa035c757f692c78ad72d69cc3c431
    Closes-Bug: #1825020
    (cherry picked from commit 8aae3e39e58a05ed50b891aeae261641f544ad1f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.