The Hyper-V driver cannot handle shared storage

Bug #1565895 reported by Lucian Petrut on 2016-04-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Lucian Petrut
compute-hyperv
Undecided
Unassigned

Bug Description

The Hyper-V driver has a few issues when attempting to use shared storage for the instance files.

Migration issues
----------------
In case of migrations, we attempt to move the instance files without checking whether shared storage is used. Note that the block_migration flag is completely ignored.

During live migrations, we let Hyper-V automatically detect whether shared storage is used or not. This seems to be troublesome when instance files reside on CSVs.

After a live migration is performed, we always try to delete the instance files. Also, if the migration fails, the destination node is not cleaned up, as the HyperVLiveMigrateData object is not used at the moment.

Image/path handling issues
--------------------------
The image cache operation uses thread locks, for which reason it is prone to race conditions when using shared storage. Switching to file locks would solve this issue.

Using UNC paths for the instance path does not work either. The driver will fail to fetch the available space. Also, it will not build the remote instance path properly when trying to move files during migrations.

tags: added: driver hyper-v
Matt Riedemann (mriedem) on 2016-04-04
tags: added: migrate
Changed in nova:
status: New → Confirmed
Changed in compute-hyperv:
status: New → Confirmed
Changed in nova:
importance: Undecided → Medium

Reviewed: https://review.openstack.org/301252
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=60c06bdd6fc006d6bff6840efdc8438de37f983d
Submitter: Jenkins
Branch: master

commit 60c06bdd6fc006d6bff6840efdc8438de37f983d
Author: Lucian Petrut <email address hidden>
Date: Mon Apr 4 17:22:29 2016 +0300

    Properly handle shared storage in case of cold migrations

    At the moment, if the destination host is other than the source
    host, we attempt to move the instance files without checking if
    shared storage is being used.

    This change fixes this issue. In case of shared storage, we'll have
    the same workflow as when the migration source is the migration
    destination as well.

    Closes-Bug: #1565895

    Change-Id: I711976b42a5de5a28de22e010b48c10d96b68ba5

Changed in compute-hyperv:
status: Confirmed → Fix Released

Reviewed: https://review.openstack.org/305744
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=3d909a68f026fd1aa928010bdde6355f6434d068
Submitter: Jenkins
Branch: stable/mitaka

commit 3d909a68f026fd1aa928010bdde6355f6434d068
Author: Lucian Petrut <email address hidden>
Date: Mon Apr 4 17:22:29 2016 +0300

    Properly handle shared storage in case of cold migrations

    At the moment, if the destination host is other than the source
    host, we attempt to move the instance files without checking if
    shared storage is being used.

    This change fixes this issue. In case of shared storage, we'll have
    the same workflow as when the migration source is the migration
    destination as well.

    Closes-Bug: #1565895

    Change-Id: I711976b42a5de5a28de22e010b48c10d96b68ba5
    (cherry picked from commit 60c06bdd6fc006d6bff6840efdc8438de37f983d)

tags: added: in-stable-mitaka
summary: - Hyper-V: cold migrations cannot handle shared storage
+ Hyper-V: migrations cannot handle shared storage
description: updated

Reviewed: https://review.openstack.org/322065
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=ba2b0bbda58228e81c163c0b7cfef29cfaf13314
Submitter: Jenkins
Branch: master

commit ba2b0bbda58228e81c163c0b7cfef29cfaf13314
Author: Lucian Petrut <email address hidden>
Date: Fri May 27 13:37:22 2016 +0300

    Fix live migration with shared storage

    At the moment, when live migration is requested, we attempt to
    copy the instance dvd disks and log files to the remote location,
    other instance files being transparently copied by Hyper-V.

    The issue is that we don't check whether shared storage is being
    used, in which case this will fail.

    This change adds this check.

    Closes-Bug: #1565895

    Change-Id: Ib646c90c830a1cd0e5401d14c6d400226b034f73

Reviewed: https://review.openstack.org/323741
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=7dcfd8c6a257ec54b97148ab87f0d7b966323761
Submitter: Jenkins
Branch: stable/mitaka

commit 7dcfd8c6a257ec54b97148ab87f0d7b966323761
Author: Lucian Petrut <email address hidden>
Date: Fri May 27 13:37:22 2016 +0300

    Fix live migration with shared storage

    At the moment, when live migration is requested, we attempt to
    copy the instance dvd disks and log files to the remote location,
    other instance files being transparently copied by Hyper-V.

    The issue is that we don't check whether shared storage is being
    used, in which case this will fail.

    This change adds this check.

    Closes-Bug: #1565895

    Change-Id: Ib646c90c830a1cd0e5401d14c6d400226b034f73
    (cherry picked from commit ba2b0bbda58228e81c163c0b7cfef29cfaf13314)

summary: - Hyper-V: migrations cannot handle shared storage
+ The Hyper-V driver cannot handle shared storage
description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/335114

Changed in nova:
assignee: nobody → Lucian Petrut (petrutlucian94)
status: Confirmed → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/347435

Fix proposed to branch: master
Review: https://review.openstack.org/347436

Fix proposed to branch: master
Review: https://review.openstack.org/347437

Matt Riedemann (mriedem) on 2016-08-02
Changed in nova:
importance: Medium → Wishlist

Reviewed: https://review.openstack.org/347435
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a4d23277338e5c917722055552e4852e0eb55c45
Submitter: Jenkins
Branch: master

commit a4d23277338e5c917722055552e4852e0eb55c45
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: update live migrate data object

    The Hyper-V driver does not properly handle shared storage at the moment.

    In order to address this, and ensure that proper cleanup is performed
    in case of failed live migrations, we'll need to update the
    HyperVLiveMigrateData object and make sure that the compute manager
    properly uses it.

    This change is part of a bigger series, attempting to fix HyperV shared
    storage related issues.

    Change-Id: I1beb56530134c83b8359ac137cdffc37a86ec8cc
    Partial-Bug: #1565895

Reviewed: https://review.openstack.org/347436
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b48ed5217aebaa1f0dd8bc7b3dd0257920e17d22
Submitter: Jenkins
Branch: master

commit b48ed5217aebaa1f0dd8bc7b3dd0257920e17d22
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: properly handle UNC instance paths

    If the configured instances dir is an UNC path such as
    \\ip\share\instances_dir, the HyperV driver fails to query the available
    disk size, as it expects a path containing a drive letter.

    Also, when attempting to move instance files during live migrations, it
    will incorrectly try to build the remote path.

    This change addresses those issues and is part of a bigger series,
    attempting to fix HyperV shared storage related issues.

    Change-Id: Ibdb9f7038bf5078867d64aef7fc63974ed8482af
    Partial-Bug: #1565895

Reviewed: https://review.openstack.org/335114
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ee7f3b30cae5f122ab85d6422a7d9dfd12ff9ab
Submitter: Jenkins
Branch: master

commit 5ee7f3b30cae5f122ab85d6422a7d9dfd12ff9ab
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: properly handle shared storage during migrations

    In case of migrations, we attempt to move the instance files without
    checking whether shared storage is used. Note that the block_migration
    flag is ignored.

    After a live migration is performed, we always try to delete the
    instance files. Also, if the migration fails, the destination node is
    not cleaned up, as the HyperVLiveMigrateData object is not used at the
    moment.

    This change addresses those issues.

    Partial-Bug: #1565895

    Change-Id: I0ac0a2d2e7a8771024a486dd5931bd05b1ecd074

description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/374966

description: updated

Reviewed: https://review.openstack.org/374966
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=7c2e267d5da59cce6eb19e1ec6e98679ec0af102
Submitter: Jenkins
Branch: master

commit 7c2e267d5da59cce6eb19e1ec6e98679ec0af102
Author: Lucian Petrut <email address hidden>
Date: Thu Sep 22 19:23:39 2016 +0300

    Fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Partial-Bug: #1565895

Reviewed: https://review.openstack.org/374857
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=e2d304810fb4eac15f73959abc3a434975904ff5
Submitter: Jenkins
Branch: master

commit e2d304810fb4eac15f73959abc3a434975904ff5
Author: Lucian Petrut <email address hidden>
Date: Thu Sep 22 15:46:45 2016 +0300

    Properly handle shared storage during live migration

    After a live migration is performed, we always try to delete the
    instance files.

    If the migration fails, the destination node is not cleaned up as
    the HyperVLiveMigrateData object is not used at the moment and the
    nova compute manager won't know whether a cleanup is needed or not.

    Also, in order to support using CSVs for storing instance files, we
    need to explicitly specify whether image files will be migrated or not
    when requesting os-win to migrate the VM.

    This change addresses those issues.

    Depends-On: I48dc6da29534bf0c477c742311ac15438d8b63ad

    Change-Id: Icf2f258ef8900b5970ff153a7b981ee53e58dd2d
    Partial-Bug: #1565895

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/400640

Change abandoned by Claudiu Belu (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/320583
Reason: shared storage is primarely used in clustered scenarios, which was introduced in mitaka.

Reviewed: https://review.openstack.org/347437
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8c40c4268be57ff2f76cf616b53992b0d32e1fd3
Submitter: Jenkins
Branch: master

commit 8c40c4268be57ff2f76cf616b53992b0d32e1fd3
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Closes-Bug: #1565895

Changed in nova:
status: In Progress → Fix Released
Vladimir Grujic (hyperbaba) wrote :

I disagree that shared storage issue fixing be abandoned on stable/liberty as many of the deployments are still running it and it is not possible to add 2016 hyperv with s2d to that clouds.

Vladimir

Reviewed: https://review.openstack.org/400636
Committed: https://git.openstack.org/cgit/openstack/compute-hyperv/commit/?id=522d0e91522ea3caa4a53d0a840020b06c3ddd31
Submitter: Jenkins
Branch: stable/mitaka

commit 522d0e91522ea3caa4a53d0a840020b06c3ddd31
Author: Lucian Petrut <email address hidden>
Date: Thu Sep 22 19:23:39 2016 +0300

    Fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Partial-Bug: #1565895
    (cherry picked from commit 7c2e267d5da59cce6eb19e1ec6e98679ec0af102)

Matt Riedemann (mriedem) wrote :

I'm not sure why I marked this as wishlist earlier, might have been a mistake.

Changed in nova:
importance: Wishlist → Medium

Reviewed: https://review.openstack.org/403251
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d4ee75c931bea7b0583db1b8d07063fb22034ec6
Submitter: Jenkins
Branch: stable/newton

commit d4ee75c931bea7b0583db1b8d07063fb22034ec6
Author: Lucian Petrut <email address hidden>
Date: Thu Jun 16 11:41:20 2016 +0300

    Hyper-V: fix image handling when shared storage is being used

    The image cache operation uses thread locks, for which reason it is
    prone to race conditions when using shared storage. This is avoided
    by using file locks, placed on the shared path.

    Change-Id: I2b002993b7ab0973960794bfdd81d4fe57bf7399
    Closes-Bug: #1565895
    (cherry picked from commit 8c40c4268be57ff2f76cf616b53992b0d32e1fd3)

tags: added: in-stable-newton

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

This issue was fixed in the openstack/nova 14.0.3 release.

Change abandoned by Claudiu Belu (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/400640
Reason: Mitaka is at EOL.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers