[LVM] lvextend command crashes with code 139

Bug #1940436 reported by Eric Harney
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Eric Harney

Bug Description

Similar to bug 1901783, the "lvextend" call needs to be wrapped to retry on code 139.

Failure seen on https://review.opendev.org/c/openstack/cinder/+/788365/10 in the tempest-integrated-storage job.

Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager Traceback (most recent call last):
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2910, in extend_volume
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager self.driver.extend_volume(volume, new_size)
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/lvm.py", line 586, in extend_volume
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager self.vg.extend_volume(volume['name'],
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/brick/local_dev/lvm.py", line 835, in extend_volume
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager self._execute(*cmd, root_helper=self._root_helper,
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager File "/usr/local/lib/python3.8/dist-packages/os_brick/executor.py", line 53, in _execute
Aug 17 22:55:25.335886 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager result = self.__execute(*args, **kwargs)
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 172, in execute
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 438, in execute
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager raise ProcessExecutionError(exit_code=_returncode,
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvextend -L 2g stack-volumes-lvmdriver-1/volume-dbedb507-ddc4-4c6d-a597-66f4e0a8dab0
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager Exit code: 139
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager Stdout: ''
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager Stderr: ' WARNING: Failed to get udev device handler for device /dev/sdd1.\n /dev/sdd15: stat failed: No such file or directory\n Path /dev/sdd15 no longer valid for device(8,63)\n /dev/sdd15: stat failed: No such file or directory\n Path /dev/sdd15 no longer valid for device(8,63)\n Device open /dev/sdd 8:48 failed errno 2\n Device open /dev/sdd 8:48 failed errno 2\n Device open /dev/sdd1 8:49 failed errno 2\n Device open /dev/sdd1 8:49 failed errno 2\n WARNING: Scan ignoring device 8:48 with no paths.\n WARNING: Scan ignoring device 8:49 with no paths.\n'
Aug 17 22:55:25.337008 ubuntu-focal-ovh-gra1-0025956218 cinder-volume[114418]: ERROR cinder.volume.manager

Eric Harney (eharney)
Changed in cinder:
assignee: nobody → Eric Harney (eharney)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/805026

Changed in cinder:
status: New → In Progress
Changed in cinder:
importance: Undecided → Medium
tags: added: gate-failure lvextend lvm
summary: - lvextend command crashes with code 139
+ [LVM] lvextend command crashes with code 139
Changed in cinder:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/805026
Committed: https://opendev.org/openstack/cinder/commit/c4b8956763c2759bd2b9fe511efac5b3edfb226d
Submitter: "Zuul (22348)"
Branch: master

commit c4b8956763c2759bd2b9fe511efac5b3edfb226d
Author: Eric Harney <email address hidden>
Date: Wed Aug 18 09:19:58 2021 -0400

    LVM: Retry lvextend commands on code 139

    Retry lvextend commands upon segfault, similar to other
    LVM calls. This affects the volume extend path.

    Change-Id: I0c0cb5308246a3dce736eade67b40be063aa78bb
    Related-Bug: #1901783
    Related-Bug: #1932188
    Closes-Bug: #1940436

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/805144

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/805144
Committed: https://opendev.org/openstack/cinder/commit/2425f3ef590524c357622748df79d87edebd17f3
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2425f3ef590524c357622748df79d87edebd17f3
Author: Eric Harney <email address hidden>
Date: Wed Aug 18 09:19:58 2021 -0400

    LVM: Retry lvextend commands on code 139

    Retry lvextend commands upon segfault, similar to other
    LVM calls. This affects the volume extend path.

    Change-Id: I0c0cb5308246a3dce736eade67b40be063aa78bb
    Related-Bug: #1901783
    Related-Bug: #1932188
    Closes-Bug: #1940436
    (cherry picked from commit c4b8956763c2759bd2b9fe511efac5b3edfb226d)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/805825

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 18.1.0

This issue was fixed in the openstack/cinder 18.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/805825
Committed: https://opendev.org/openstack/cinder/commit/a7c34584af53b01bd359e1df37e3704288ebe30e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a7c34584af53b01bd359e1df37e3704288ebe30e
Author: Eric Harney <email address hidden>
Date: Wed Aug 18 09:19:58 2021 -0400

    LVM: Retry lvextend commands on code 139

    Retry lvextend commands upon segfault, similar to other
    LVM calls. This affects the volume extend path.

    Change-Id: I0c0cb5308246a3dce736eade67b40be063aa78bb
    Related-Bug: #1901783
    Related-Bug: #1932188
    Closes-Bug: #1940436
    (cherry picked from commit c4b8956763c2759bd2b9fe511efac5b3edfb226d)
    (cherry picked from commit 2425f3ef590524c357622748df79d87edebd17f3)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 19.0.0.0rc1

This issue was fixed in the openstack/cinder 19.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 17.2.0

This issue was fixed in the openstack/cinder 17.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.