The Hyper-V driver cannot handle shared storage

[Lucian Petrut]

The Hyper-V driver has a few issues when attempting to use shared storage for the instance files.

Migration issues
----------------
In case of migrations, we attempt to move the instance files without checking whether shared storage is used. Note that the block_migration flag is completely ignored.

During live migrations, we let Hyper-V automatically detect whether shared storage is used or not. This seems to be troublesome when instance files reside on CSVs.

After a live migration is performed, we always try to delete the instance files. Also, if the migration fails, the destination node is not cleaned up, as the HyperVLiveMigrateData object is not used at the moment.

Image/path handling issues
--------------------------
The image cache operation uses thread locks, for which reason it is prone to race conditions when using shared storage. Switching to file locks would solve this issue.

Using UNC paths for the instance path does not work either. The driver will fail to fetch the available space. Also, it will not build the remote instance path properly when trying to move files during migrations.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/301252
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=60c06bdd6fc006d6bff6840efdc8438de37f983d
Submitter: Jenkins
Branch: master

commit 60c06bdd6fc006d6bff6840efdc8438de37f983d
Author: Lucian Petrut <email address hidden>
Date: Mon Apr 4 17:22:29 2016 +0300

    Properly handle shared storage in case of cold migrations

    At the moment, if the destination host is other than the source
    host, we attempt to move the instance files without checking if
    shared storage is being used.

    This change fixes this issue. In case of shared storage, we'll have
    the same workflow as when the migration source is the migration
    destination as well.

    Closes-Bug: #1565895

    Change-Id: I711976b42a5de5a28de22e010b48c10d96b68ba5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/305744

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/305744
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=3d909a68f026fd1aa928010bdde6355f6434d068
Submitter: Jenkins
Branch: stable/mitaka

commit 3d909a68f026fd1aa928010bdde6355f6434d068
Author: Lucian Petrut <email address hidden>
Date: Mon Apr 4 17:22:29 2016 +0300

    Properly handle shared storage in case of cold migrations

    At the moment, if the destination host is other than the source
    host, we attempt to move the instance files without checking if
    shared storage is being used.

    This change fixes this issue. In case of shared storage, we'll have
    the same workflow as when the migration source is the migration
    destination as well.

    Closes-Bug: #1565895

    Change-Id: I711976b42a5de5a28de22e010b48c10d96b68ba5
    (cherry picked from commit 60c06bdd6fc006d6bff6840efdc8438de37f983d)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/320583

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/322065
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=ba2b0bbda58228e81c163c0b7cfef29cfaf13314
Submitter: Jenkins
Branch: master

commit ba2b0bbda58228e81c163c0b7cfef29cfaf13314
Author: Lucian Petrut <email address hidden>
Date: Fri May 27 13:37:22 2016 +0300

    Fix live migration with shared storage

    At the moment, when live migration is requested, we attempt to
    copy the instance dvd disks and log files to the remote location,
    other instance files being transparently copied by Hyper-V.

    The issue is that we don't check whether shared storage is being
    used, in which case this will fail.

    This change adds this check.

    Closes-Bug: #1565895

    Change-Id: Ib646c90c830a1cd0e5401d14c6d400226b034f73

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/323741

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/323741
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=7dcfd8c6a257ec54b97148ab87f0d7b966323761
Submitter: Jenkins
Branch: stable/mitaka

commit 7dcfd8c6a257ec54b97148ab87f0d7b966323761
Author: Lucian Petrut <email address hidden>
Date: Fri May 27 13:37:22 2016 +0300

    Fix live migration with shared storage

    At the moment, when live migration is requested, we attempt to
    copy the instance dvd disks and log files to the remote location,
    other instance files being transparently copied by Hyper-V.

    The issue is that we don't check whether shared storage is being
    used, in which case this will fail.

    This change adds this check.

    Closes-Bug: #1565895

    Change-Id: Ib646c90c830a1cd0e5401d14c6d400226b034f73
    (cherry picked from commit ba2b0bbda58228e81c163c0b7cfef29cfaf13314)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/335114

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/347435

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/347436

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/347437

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347435
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a4d23277338e5c917722055552e4852e0eb55c45
Submitter: Jenkins
Branch: master

commit a4d23277338e5c917722055552e4852e0eb55c45
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: update live migrate data object

    The Hyper-V driver does not properly handle shared storage at the moment.

    In order to address this, and ensure that proper cleanup is performed
    in case of failed live migrations, we'll need to update the
    HyperVLiveMigrateData object and make sure that the compute manager
    properly uses it.

    This change is part of a bigger series, attempting to fix HyperV shared
    storage related issues.

    Change-Id: I1beb56530134c83b8359ac137cdffc37a86ec8cc
    Partial-Bug: #1565895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347436
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b48ed5217aebaa1f0dd8bc7b3dd0257920e17d22
Submitter: Jenkins
Branch: master

commit b48ed5217aebaa1f0dd8bc7b3dd0257920e17d22
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: properly handle UNC instance paths

    If the configured instances dir is an UNC path such as
    \\ip\share\instances_dir, the HyperV driver fails to query the available
    disk size, as it expects a path containing a drive letter.

    Also, when attempting to move instance files during live migrations, it
    will incorrectly try to build the remote path.

    This change addresses those issues and is part of a bigger series,
    attempting to fix HyperV shared storage related issues.

    Change-Id: Ibdb9f7038bf5078867d64aef7fc63974ed8482af
    Partial-Bug: #1565895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/335114
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ee7f3b30cae5f122ab85d6422a7d9dfd12ff9ab
Submitter: Jenkins
Branch: master

commit 5ee7f3b30cae5f122ab85d6422a7d9dfd12ff9ab
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: properly handle shared storage during migrations

    In case of migrations, we attempt to move the instance files without
    checking whether shared storage is used. Note that the block_migration
    flag is ignored.

    After a live migration is performed, we always try to delete the
    instance files. Also, if the migration fails, the destination node is
    not cleaned up, as the HyperVLiveMigrateData object is not used at the
    moment.

    This change addresses those issues.

    Partial-Bug: #1565895

    Change-Id: I0ac0a2d2e7a8771024a486dd5931bd05b1ecd074

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/374857

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/374966

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374966
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=7c2e267d5da59cce6eb19e1ec6e98679ec0af102
Submitter: Jenkins
Branch: master

commit 7c2e267d5da59cce6eb19e1ec6e98679ec0af102
Author: Lucian Petrut <email address hidden>
Date: Thu Sep 22 19:23:39 2016 +0300

    Fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Partial-Bug: #1565895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/380448

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374857
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=e2d304810fb4eac15f73959abc3a434975904ff5
Submitter: Jenkins
Branch: master

commit e2d304810fb4eac15f73959abc3a434975904ff5
Author: Lucian Petrut <email address hidden>
Date: Thu Sep 22 15:46:45 2016 +0300

    Properly handle shared storage during live migration

    After a live migration is performed, we always try to delete the
    instance files.

    If the migration fails, the destination node is not cleaned up as
    the HyperVLiveMigrateData object is not used at the moment and the
    nova compute manager won't know whether a cleanup is needed or not.

    Also, in order to support using CSVs for storing instance files, we
    need to explicitly specify whether image files will be migrated or not
    when requesting os-win to migrate the VM.

    This change addresses those issues.

    Depends-On: I48dc6da29534bf0c477c742311ac15438d8b63ad

    Change-Id: Icf2f258ef8900b5970ff153a7b981ee53e58dd2d
    Partial-Bug: #1565895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/400636

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/400640

[OpenStack Infra (hudson-openstack)]

Change abandoned by Claudiu Belu (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/320583
Reason: shared storage is primarely used in clustered scenarios, which was introduced in mitaka.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347437
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8c40c4268be57ff2f76cf616b53992b0d32e1fd3
Submitter: Jenkins
Branch: master

commit 8c40c4268be57ff2f76cf616b53992b0d32e1fd3
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Closes-Bug: #1565895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/403251

[Vladimir Grujic (hyperbaba)]

I disagree that shared storage issue fixing be abandoned on stable/liberty as many of the deployments are still running it and it is not possible to add 2016 hyperv with s2d to that clouds.

Vladimir

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/400636
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=522d0e91522ea3caa4a53d0a840020b06c3ddd31
Submitter: Jenkins
Branch: stable/mitaka

commit 522d0e91522ea3caa4a53d0a840020b06c3ddd31
Author: Lucian Petrut <email address hidden>
Date: Thu Sep 22 19:23:39 2016 +0300

    Fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Partial-Bug: #1565895
    (cherry picked from commit 7c2e267d5da59cce6eb19e1ec6e98679ec0af102)

[Matt Riedemann (mriedem)]

I'm not sure why I marked this as wishlist earlier, might have been a mistake.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/403251
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d4ee75c931bea7b0583db1b8d07063fb22034ec6
Submitter: Jenkins
Branch: stable/newton

commit d4ee75c931bea7b0583db1b8d07063fb22034ec6
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Closes-Bug: #1565895
    (cherry picked from commit 8c40c4268be57ff2f76cf616b53992b0d32e1fd3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.3 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Claudiu Belu (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/400640
Reason: Mitaka is at EOL.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380448
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=df7d68e9ac2e5504db992f50b57ea450ae4741cb
Submitter: Zuul
Branch: master

commit df7d68e9ac2e5504db992f50b57ea450ae4741cb
Author: Lucian Petrut <email address hidden>
Date: Fri Sep 30 19:52:08 2016 +0300

    Hyper-V: fix live migration with CSVs

    In order to support live migrating instances placed on Cluster
    Shared Volumes, we must explicitly tell os-win whether disks will
    be migrated or not, ensuring that the right live migration type
    will be requested.

    Otherwise, the live migration will fail, Hyper-V complaining that
    it could not move the instance disks (although this operation would
    not be necessary).

    Note that this was not a problem with other forms of shared storage
    (for example having instance files on a SMB share), in which case
    Hyper-V automatically detects that shared storage is being used.

    Change-Id: Ic1949d48fe0de73a3de9417bc8462051e702a298
    Partial-Bug: #1565895