instance.uuid no longer being a str breaks powervm scsi disconnect

Bug #1766692 reported by Eric Fried
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Eric Fried
nova-powervm
Fix Released
Undecided
Eric Fried
pypowervm
Fix Released
Critical
Eric Fried

Bug Description

Long-standing code in pypowervm [1] used isinstance(..., str) to help identify whether an input was a UUID or an integer short ID. This is used with to find SCSI mappings [2] with Instance.uuid [3] when disconnecting a disk during destroy [4].

Then this change in oslo.versionedobjects happened [5], resulting in instance.uuid no longer being a str. So the check at [1] fails, and we try to int() a UUID string, resulting in [6], pasted here in case that link expires:

PowerVM error destroying instance.: ValueError: invalid literal for int() with base 10: '4E27E1E6-6A24-4F0A-8E7B-2BBE7B4A28BA'
 Traceback (most recent call last):
   File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/driver.py", line 672, in _destroy
     _setup_flow_and_run()
   File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/driver.py", line 668, in _setup_flow_and_run
     tf_base.run(flow, instance=instance)
   File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/tasks/base.py", line 40, in run
     return eng.run()
   File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
     for _state in self.run_iter(timeout=timeout):
   File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
     failure.Failure.reraise_if_any(er_failures)
   File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 336, in reraise_if_any
     failures[0].reraise()
   File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 343, in reraise
     six.reraise(*self._exc_info)
   File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
     result = task.execute(**arguments)
   File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/tasks/storage.py", line 452, in execute
     self.instance, stg_ftsk=self.stg_ftsk, disk_type=self.disk_type)
   File "/opt/stack/nova-powervm/nova_powervm/virt/powervm/disk/ssp.py", line 174, in disconnect_disk
     match_func=match_func)
   File "/usr/local/lib/python2.7/dist-packages/pypowervm/tasks/scsi_mapper.py", line 503, in find_maps
     is_uuid, client_id = uuid.id_or_uuid(client_lpar_id)
   File "/usr/local/lib/python2.7/dist-packages/pypowervm/utils/uuid.py", line 55, in id_or_uuid
     ret_id = int(an_id)
 ValueError: invalid literal for int() with base 10: '4E27E1E6-6A24-4F0A-8E7B-2BBE7B4A28BA'

(Okay, our bad for using str at all - though to be fair, it's possible that code predates the very existence of py3.)

The right fix is for [1] to use is_uuid_like from oslo.utils. This shall be done. However, since [5] was backported to queens and pike, unless we can get away with backporting requirements changes, we may have to do something backportable in nova-powervm and nova as well.

[1] https://github.com/powervm/pypowervm/blob/release/1.1.14/pypowervm/utils/uuid.py#L50
[2] https://github.com/powervm/pypowervm/blob/release/1.1.14/pypowervm/tasks/scsi_mapper.py#L503
[3] https://github.com/openstack/nova/blob/1605391084d6a1f57384ef48bad8ca2185cf6fa7/nova/virt/powervm/disk/ssp.py#L128
[4] https://github.com/openstack/nova/blob/1605391084d6a1f57384ef48bad8ca2185cf6fa7/nova/virt/powervm/driver.py#L272
[5] https://review.openstack.org/#/q/Ic6b6308fb1960ec40407e6efde30137b64543e72
[6] http://184.172.12.213/58/557958/10/check/nova-out-of-tree-pvm/c1d7e99/logs/n-cpu.txt.gz?#_Apr_20_08_51_16_452651

Eric Fried (efried)
Changed in pypowervm:
status: New → In Progress
assignee: nobody → Eric Fried (efried)
importance: Undecided → Critical
Revision history for this message
Eric Fried (efried) wrote :

The revert is here: https://review.openstack.org/#/q/I178f14cdc670d7a696778891e587ef75de208fc2

Hopefully they will avoid the broken version in the stable branches of the requirements project.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/564276

Changed in nova:
assignee: nobody → Eric Fried (efried)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-powervm (master)

Reviewed: https://review.openstack.org/564278
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=af0e814d628289c07c8e88629539a3e42c3d3daa
Submitter: Zuul
Branch: master

commit af0e814d628289c07c8e88629539a3e42c3d3daa
Author: Eric Fried <email address hidden>
Date: Wed Apr 25 11:36:32 2018 -0500

    Bump pypowervm minimum to 1.1.15

    The referenced bug is fixed in pypowervm 1.1.15. Without the fix,
    PowerVM device detach during destroy (and therefore PowerVM CI) is
    broken.

    Change-Id: I105e4b7001236d262998322273dcdd8e90d0b8c5
    Depends-On: https://review.openstack.org/564275
    Closes-Bug: #1766692

Changed in nova-powervm:
status: New → Fix Released
Changed in pypowervm:
status: In Progress → Fix Released
Eric Fried (efried)
Changed in nova-powervm:
assignee: nobody → Eric Fried (efried)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/564276
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=864acf7c8db7c9a25a919b7ad698f0d078a4dec0
Submitter: Zuul
Branch: master

commit 864acf7c8db7c9a25a919b7ad698f0d078a4dec0
Author: Eric Fried <email address hidden>
Date: Wed Apr 25 11:34:22 2018 -0500

    Bump pypowervm minimum to 1.1.15

    The referenced bug is fixed in pypowervm 1.1.15. Without the fix,
    PowerVM device detach during destroy (and therefore PowerVM CI) is
    broken.

    Change-Id: Icaabadfb17dd87207d99938224713a78dc925674
    Depends-On: https://review.openstack.org/#/c/564275/
    Closes-Bug: #1766692

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova-powervm (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/567434

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/567599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova-powervm (stable/queens)

Reviewed: https://review.openstack.org/567434
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=49330fcf28425cebfb281d0721a2d1d0bcf4d801
Submitter: Zuul
Branch: stable/queens

commit 49330fcf28425cebfb281d0721a2d1d0bcf4d801
Author: Eric Fried <email address hidden>
Date: Fri Apr 20 16:09:02 2018 -0500

    Stringify instance UUID

    oslo.versionedobjects merged a patch [1] resulting in the referenced
    bug. A second patch [2] was merged that was supposed to fix the issue
    If coerce's value arg is a unicode string, then the method will return
    unicode [3]. Subsequent checks for isinstance(str) [4] will fail and a
    TypeError will be raised trying to handle the uuid as an int in the
    else block. This change ensures that the uuid is an instance of str so
    that it is handled properly. This problem was fixed in pypowervm 1.1.15
    [5] and the minimum version was bumped for master. However, we can't
    backport the pypowervm requirements bump to the stable branches so we
    have to fix it here.

    [1] https://review.openstack.org/#/c/559815/
    [2] https://review.openstack.org/#/c/561674/
    [3] https://review.openstack.org/#/c/561674/2/oslo_versionedobjects/fields.py@367
    [4] https://github.com/powervm/pypowervm/blob/1.1.10/pypowervm/utils/uuid.py#L50-L56
    [5] https://github.com/powervm/pypowervm/commit/d55b4c84

    Related-Bug: #1766692
    Change-Id: Ic3b11b071a055177bbcfb0555e2a391ba419d191

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova-powervm (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/567616

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova-powervm (stable/pike)

Reviewed: https://review.openstack.org/567616
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=d051c80a74f37f9264a55480dfbab03769194b2f
Submitter: Zuul
Branch: stable/pike

commit d051c80a74f37f9264a55480dfbab03769194b2f
Author: Eric Fried <email address hidden>
Date: Fri Apr 20 16:09:02 2018 -0500

    Stringify instance UUID

    oslo.versionedobjects merged a patch [1] resulting in the referenced
    bug. A second patch [2] was merged that was supposed to fix the issue
    If coerce's value arg is a unicode string, then the method will return
    unicode [3]. Subsequent checks for isinstance(str) [4] will fail and a
    TypeError will be raised trying to handle the uuid as an int in the
    else block. This change ensures that the uuid is an instance of str so
    that it is handled properly. This problem was fixed in pypowervm 1.1.15
    [5] and the minimum version was bumped for master. However, we can't
    backport the pypowervm requirements bump to the stable branches so we
    have to fix it here.

    [1] https://review.openstack.org/#/c/559815/
    [2] https://review.openstack.org/#/c/561674/
    [3] https://review.openstack.org/#/c/561674/2/oslo_versionedobjects/fields.py@367
    [4] https://github.com/powervm/pypowervm/blob/1.1.6/pypowervm/utils/uuid.py#L50-L56
    [5] https://github.com/powervm/pypowervm/commit/d55b4c84

    Related-Bug: #1766692
    Change-Id: Ic3b11b071a055177bbcfb0555e2a391ba419d191
    (cherry picked from commit 49330fcf28425cebfb281d0721a2d1d0bcf4d801)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/567599
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2386e015ef702ae8db29e396e436a9e293df8f3b
Submitter: Zuul
Branch: stable/queens

commit 2386e015ef702ae8db29e396e436a9e293df8f3b
Author: esberglu <email address hidden>
Date: Thu May 10 09:08:14 2018 -0500

    Stringify instance UUID

    oslo.versionedobjects merged a patch [1] resulting in the referenced
    bug. A second patch [2] was merged that was supposed to fix the issue.
    If coerce's value arg is a unicode string, then the method will return
    unicode [3]. Subsequent checks for isinstance(str) [4] will fail and a
    TypeError will be raised trying to handle the uuid as an int in the
    else block. This change ensures that the uuid is an instance of str so
    that it is handled properly. This problem was fixed in pypowervm 1.1.15
    [5] and the minimum version was bumped for master. However, we can't
    backport the pypowervm requirements bump to the stable branches so we
    have to fix it here.

    [1] https://review.openstack.org/#/c/559815/
    [2] https://review.openstack.org/#/c/561674/
    [3] https://review.openstack.org/#/c/561674/2/oslo_versionedobjects/fields.py@367
    [4] https://github.com/powervm/pypowervm/blob/1.1.10/pypowervm/utils/uuid.py#L50-L56
    [5] https://github.com/powervm/pypowervm/commit/d55b4c84

    Change-Id: I1baef962b4b8074f3f9b9ad5402970edc9a3a776
    Related-Bug: #1766692

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/569421

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/569421
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=320132d04f3c469dbe845d5a31c0c004c63f2876
Submitter: Zuul
Branch: stable/pike

commit 320132d04f3c469dbe845d5a31c0c004c63f2876
Author: esberglu <email address hidden>
Date: Thu May 10 09:08:14 2018 -0500

    Stringify instance UUID

    oslo.versionedobjects merged a patch [1] resulting in the referenced
    bug. A second patch [2] was merged that was supposed to fix the issue.
    If coerce's value arg is a unicode string, then the method will return
    unicode [3]. Subsequent checks for isinstance(str) [4] will fail and a
    TypeError will be raised trying to handle the uuid as an int in the
    else block. This change ensures that the uuid is an instance of str so
    that it is handled properly. This problem was fixed in pypowervm 1.1.15
    [5] and the minimum version was bumped for master. However, we can't
    backport the pypowervm requirements bump to the stable branches so we
    have to fix it here.

    [1] https://review.openstack.org/#/c/559815/
    [2] https://review.openstack.org/#/c/561674/
    [3] https://review.openstack.org/#/c/561674/2/oslo_versionedobjects/fields.py@367
    [4] https://github.com/powervm/pypowervm/blob/1.1.10/pypowervm/utils/uuid.py#L50-L56
    [5] https://github.com/powervm/pypowervm/commit/d55b4c84

    Change-Id: I1baef962b4b8074f3f9b9ad5402970edc9a3a776
    Related-Bug: #1766692
    (cherry picked from commit 2386e015ef702ae8db29e396e436a9e293df8f3b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova-powervm 7.0.0.0b2

This issue was fixed in the openstack/nova-powervm 7.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0b2

This issue was fixed in the openstack/nova 18.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.