libvirtError exceptions during volume attach leave volume connected to host

Bug #1826523 reported by Lee Yarwood on 2019-04-26
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Lee Yarwood
Queens
Medium
Lee Yarwood
Rocky
Medium
Lee Yarwood
Stein
Medium
Lee Yarwood
Ubuntu Cloud Archive
Status tracked in Stein
Queens
Medium
Unassigned
Rocky
Medium
Unassigned
Stein
Medium
Unassigned
nova (Ubuntu)
Medium
Sahid Orentino
Bionic
Medium
Sahid Orentino
Cosmic
Medium
Sahid Orentino
Disco
Medium
Sahid Orentino

Bug Description

[Impact]

 * This is an additional patch required for bug #1825882, when
a libvirt exception that prevents the volume attachment to complete,
the underlying volumes should be disconnected from the host.

[Test Case]

* Deploy any OpenStack version up to Pike , which includes ceph backed cinder
* Create a guest VM (openstack server ...)
* Create a test cinder volume

$ openstack volume create test --size 10

* Force a drop on ceph traffic. Run the following command on the nova hypervisor on which the server runs.

$ iptables -A OUTPUT -d ceph-mon-addr -p tcp --dport 6800 -j DROP

* Attach the volume to a running instance.

$ openstack server add volume 7151f507-a6b7-4f6d-a4cc-fd223d9feb5d 742ff117-21ae-4d1b-a52b-5b37955716ff

* This should cause the volume attachment to fail

$ virsh domblklist instance-xxxxx
Target Source
------------------------------------------------
vda nova/7151f507-a6b7-4f6d-a4cc-fd223d9feb5d_disk

No volume should attached after this step.

* If the behavior is fixed:

   * Check that openstack server show , doesn't displays the displays the volume as attached.

* If the behavior isn't fixed:

   * openstack server show <ID> , will display the volume in the volumes_attached property.

[Expected result]

* Volume attach fails and the volume is disconnected from the host.

[Actual result]

* Volume attach fails but remains connected to the host.

[Regression Potential]

* The patches have been cherry-picked from upstream which helps to reduce the regression potential of these fixes.

[Other Info]

* N/A

Matt Riedemann (mriedem) wrote :
Changed in nova:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Lee Yarwood (lyarwood)
tags: added: libvirt volumes

Reviewed: https://review.opendev.org/655712
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=091a910576d9b580678f1881fffa425ab4632f48
Submitter: Zuul
Branch: master

commit 091a910576d9b580678f1881fffa425ab4632f48
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/657109
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=048d5b790f3da2756a0a1bf2bc015812cb24d53a
Submitter: Zuul
Branch: stable/stein

commit 048d5b790f3da2756a0a1bf2bc015812cb24d53a
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213
    (cherry picked from commit 091a910576d9b580678f1881fffa425ab4632f48)

Reviewed: https://review.opendev.org/657110
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=786083c91fa69ba9ff90bafe80d3b83b9f9bbc69
Submitter: Zuul
Branch: stable/rocky

commit 786083c91fa69ba9ff90bafe80d3b83b9f9bbc69
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213
    (cherry picked from commit 091a910576d9b580678f1881fffa425ab4632f48)
    (cherry picked from commit 048d5b790f3da2756a0a1bf2bc015812cb24d53a)

Reviewed: https://review.opendev.org/657111
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=99a63a6633f623cc4e6c9b361965a9c0935113cf
Submitter: Zuul
Branch: stable/queens

commit 99a63a6633f623cc4e6c9b361965a9c0935113cf
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213
    (cherry picked from commit 091a910576d9b580678f1881fffa425ab4632f48)
    (cherry picked from commit 048d5b790f3da2756a0a1bf2bc015812cb24d53a)
    (cherry picked from commit 786083c91fa69ba9ff90bafe80d3b83b9f9bbc69)

description: updated
tags: added: sts
Changed in nova (Ubuntu):
status: New → Triaged
Changed in nova (Ubuntu Disco):
status: New → Triaged
Changed in nova (Ubuntu):
importance: Undecided → Medium
Changed in nova (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in nova (Ubuntu Disco):
importance: Undecided → Medium
Changed in nova (Ubuntu Bionic):
importance: Undecided → Medium
status: New → Triaged
Changed in nova (Ubuntu Cosmic):
status: New → Triaged
Changed in nova (Ubuntu):
assignee: nobody → Sahid Orentino (sahid-ferdjaoui)
Changed in nova (Ubuntu Bionic):
assignee: nobody → Sahid Orentino (sahid-ferdjaoui)
Changed in nova (Ubuntu Cosmic):
assignee: nobody → Sahid Orentino (sahid-ferdjaoui)
Changed in nova (Ubuntu Disco):
assignee: nobody → Sahid Orentino (sahid-ferdjaoui)
description: updated

Hello Lee, or anyone else affected,

Accepted nova into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:19.0.0-0ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in nova (Ubuntu Disco):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-disco
Łukasz Zemczak (sil2100) wrote :

Hello Lee, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.1.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in nova (Ubuntu Cosmic):
status: Triaged → Fix Committed
tags: added: verification-needed-cosmic
Corey Bryant (corey.bryant) wrote :

Hello Lee, or anyone else affected,

Accepted nova into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:19.0.0-0ubuntu5

---------------
nova (2:19.0.0-0ubuntu5) eoan; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:41:27 +0200

Changed in nova (Ubuntu):
status: Triaged → Fix Released
Corey Bryant (corey.bryant) wrote :

Hello Lee, or anyone else affected,

Accepted nova into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-rocky-needed
Łukasz Zemczak (sil2100) wrote :

Hello Lee, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.9-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in nova (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed-bionic
Corey Bryant (corey.bryant) wrote :

Hello Lee, or anyone else affected,

Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-queens-needed
Jorge Niedbalski (niedbalski) wrote :

I have verified that 2:19.0.0-0ubuntu2.3 fixes the issue on disco,
raising the following exception:

 libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)

No attachment has been found at the device after this.

Jorge Niedbalski (niedbalski) wrote :

I have verified the following versions:

* bionic: 17.0.9-0ubuntu3
* cosmic: 18.1.0-0ubuntu3
* disco: 19.0.0-0ubuntu2.3

The following exception is raised when the traffic to the ceph-mon is dropped by
iptables.

 libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)

No attachment has been found at the vms after this operation fails, no volume added to
volumes_attached property.

tags: added: verification-done-bionic verification-done-cosmic verification-done-disco
removed: verification-needed-bionic verification-needed-cosmic verification-needed-disco
tags: added: sts-sru-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:17.0.9-0ubuntu3

---------------
nova (2:17.0.9-0ubuntu3) bionic; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:17.0.9-0ubuntu2) bionic; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).
  * d/p/skip-double-word-hacking-test.patch: Cherry-picked from upstream
    to work-around test_hacking failures with new versions of python
    (LP: #1782786).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 11:06:11 +0200

Changed in nova (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:18.1.0-0ubuntu3

---------------
nova (2:18.1.0-0ubuntu3) cosmic; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:18.1.0-0ubuntu2) cosmic; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:58:45 +0200

Changed in nova (Ubuntu Cosmic):
status: Fix Committed → Fix Released

This issue was fixed in the openstack/nova 19.0.1 release.

Jorge Niedbalski (niedbalski) wrote :

OK, verified S/Q/R

The following exception is raised when the traffic to the ceph-mon is dropped by
iptables.

** libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)
** No attachment has been found at the vms after this operation fails, no volume added to
volumes_attached property. (https://pastebin.canonical.com/p/kkmRcfrsjT/)

tags: added: sts-sru-done verification-done verification-queens-done verification-rocky-done verification-stein-done
removed: sts-sru-needed verification-needed verification-queens-needed verification-rocky-needed verification-stein-needed

This issue was fixed in the openstack/nova 18.2.1 release.

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package nova - 2:19.0.0-0ubuntu2.3~cloud0
---------------

 nova (2:19.0.0-0ubuntu2.3~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:19.0.0-0ubuntu2.3) disco; urgency=medium
 .
   * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
     virsh disk attach does not fail silently (LP: #1825882).
   * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
     always disconnect volumes after libvirt exceptions (LP: #1826523).

James Page (james-page) wrote :

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package nova - 2:17.0.9-0ubuntu3~cloud0
---------------

 nova (2:17.0.9-0ubuntu3~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:17.0.9-0ubuntu3) bionic; urgency=medium
 .
   * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
     virsh disk attach does not fail silently (LP: #1825882).
   * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
     always disconnect volumes after libvirt exceptions (LP: #1826523).

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:19.0.0-0ubuntu2.3

---------------
nova (2:19.0.0-0ubuntu2.3) disco; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:19.0.0-0ubuntu2.2) disco; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

nova (2:19.0.0-0ubuntu2.1) disco; urgency=medium

  * d/gbp.conf: Create stable/stein branch.
  * d/p/eventlet-monkey-patching-should-be-as-early-as-possible.patch:
    Cherry-picked from upstream stable/stein review to fix py3+wsgi+ssl crash
    (LP: #1808951).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:54:46 +0200

Changed in nova (Ubuntu Disco):
status: Fix Committed → Fix Released

This issue was fixed in the openstack/nova 17.0.11 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers