A shelved_offload VM's volumes are still attached to a host

[Shoham Peller]

When shelve_offloading a VM, the VM loses it's connection to a host.
However, connection to the host is not terminated to it's volumes, so they are still attached to a host.

Afterwards, when the VM is unshleved, nova calls initialize_connection to the new host for it's volumes, and they are now connected to 2 hosts.

The correct behaviour is to call terminate_connection on the VM's volumes when it's being shelved_offloaded

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/281990

[OpenStack Infra (hudson-openstack)]

Change abandoned by Shoham Peller (<email address hidden>) on branch: master
Review: https://review.openstack.org/281990
Reason: Duplicates https://review.openstack.org/#/c/257275

[Andrea Rosa (andrea-rosa-m)]

To properly fix this bug we depends on the resolution of a cinder bug, precisely:
https://bugs.launchpad.net/cinder/+bug/1527278
It is not enough to call the terminate_connection in Nova, that call doesn't close correctly the open connections on the Cinder side, we need to call a different method "remove_export" which is not exposed bu the API yet.

[Andrea Rosa (andrea-rosa-m)]

The resolution of this bug depends on a fix in cinder. I am not actively working on the cinder fix so this bug is stuck at the moment.

[Sivasathurappan Radhakrishnan (siva-radhakrishnan)]

changing it to confirmed to avoid any confusion on the status of the bug

[Shoham Peller (shoham-peller)]

@andrea-rosa-m
Isn't calling terminate_connection better than nothing? This makes volumes be connected to more than 1 node in cinder.
Why not add the terminate-connection, and then add the remove_export, once it is available?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/257275
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the nova review queue sane. Please feel free to restore the change if you're still working on it.

[Sean Dague (sdague)]

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/257275 in branch: master

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/257275
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257275
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e89e1bdc60211622440c964f8be8563da89341ac
Submitter: Jenkins
Branch: master

commit e89e1bdc60211622440c964f8be8563da89341ac
Author: Andrea Rosa <email address hidden>
Date: Thu Sep 14 13:47:06 2017 -0400

    Call terminate_connection when shelve_offloading

    When nova performs a shelve offload for an instance, it needs to terminate
    all the volume connections for that instance as with the shelve offload
    it is not guaranteed that the instance will be placed on the same host once
    it gets unshelved.
    This change adds the call to the terminate_volume_connections on the
    _shelve_offload_instance method in the compute manager.

    Closes-Bug: #1547142

    Change-Id: I8849ae0f54605e003d5b294ca3d66dcef89d7d27

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/504270

[Matt Riedemann (mriedem)]

To come back to some earlier discussion about this bug, we discussed this with the cinder team at the queens PTG and it is OK to call terminate connection during shelve offload, notes are here:

https://etherpad.openstack.org/p/cinder-ptg-queens

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/504273

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/504270
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8365eb6cb987c834b1f35c04be13aa97db36a4a1
Submitter: Jenkins
Branch: stable/pike

commit 8365eb6cb987c834b1f35c04be13aa97db36a4a1
Author: Andrea Rosa <email address hidden>
Date: Thu Sep 14 13:47:06 2017 -0400

    Call terminate_connection when shelve_offloading

    When nova performs a shelve offload for an instance, it needs to terminate
    all the volume connections for that instance as with the shelve offload
    it is not guaranteed that the instance will be placed on the same host once
    it gets unshelved.
    This change adds the call to the terminate_volume_connections on the
    _shelve_offload_instance method in the compute manager.

    Closes-Bug: #1547142

    Change-Id: I8849ae0f54605e003d5b294ca3d66dcef89d7d27
    (cherry picked from commit e89e1bdc60211622440c964f8be8563da89341ac)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/504273
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=286fa12ad128ad22d2e9c5002c54dfdb54faac16
Submitter: Zuul
Branch: stable/ocata

commit 286fa12ad128ad22d2e9c5002c54dfdb54faac16
Author: Andrea Rosa <email address hidden>
Date: Thu Sep 14 13:47:06 2017 -0400

    Call terminate_connection when shelve_offloading

    When nova performs a shelve offload for an instance, it needs to terminate
    all the volume connections for that instance as with the shelve offload
    it is not guaranteed that the instance will be placed on the same host once
    it gets unshelved.
    This change adds the call to the terminate_volume_connections on the
    _shelve_offload_instance method in the compute manager.

    Closes-Bug: #1547142

    Conflicts:
          nova/tests/unit/compute/test_shelve.py

    NOTE(mriedem): The conflicts in the test are just due to not having
    resource allocation cleanup for placement in shelve offload in Ocata.

    Change-Id: I8849ae0f54605e003d5b294ca3d66dcef89d7d27
    (cherry picked from commit e89e1bdc60211622440c964f8be8563da89341ac)
    (cherry picked from commit 8365eb6cb987c834b1f35c04be13aa97db36a4a1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.0 release.