| 2016-01-19 20:52:19 |
Rafael David Tinoco |
bug |
|
|
added bug |
| 2016-01-19 20:52:37 |
Rafael David Tinoco |
bug |
|
|
added subscriber Mathieu Trudel-Lapierre |
| 2016-01-19 20:52:49 |
Rafael David Tinoco |
bug |
|
|
added subscriber Louis Bouchard |
| 2016-01-19 20:53:15 |
Rafael David Tinoco |
bug |
|
|
added subscriber Gábor Mészáros |
| 2016-01-19 20:54:01 |
Rafael David Tinoco |
attachment added |
|
iscsiclient.sh https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1535898/+attachment/4552892/+files/iscsiclient.sh |
|
| 2016-01-19 20:54:12 |
Rafael David Tinoco |
attachment added |
|
iscsiserver.sh https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1535898/+attachment/4552893/+files/iscsiserver.sh |
|
| 2016-01-19 20:55:08 |
Rafael David Tinoco |
attachment added |
|
multipath.conf https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1535898/+attachment/4552894/+files/multipath.conf |
|
| 2016-01-19 20:55:47 |
Rafael David Tinoco |
attachment added |
|
iscsi.txt https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1535898/+attachment/4552895/+files/iscsi.txt |
|
| 2016-01-19 20:57:08 |
Rafael David Tinoco |
attachment added |
|
_sbin_multipathd.0.crash https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1535898/+attachment/4552896/+files/_sbin_multipathd.0.crash |
|
| 2016-01-19 21:04:26 |
Rafael David Tinoco |
description |
We have a problem on multipath-tools. I created 2 hosts:
iscsi-server
iscsi-client
With 4 NICs in between them and with a simple multibus multipath.
With that I was able to check that there is a regression in multipath-tools.
It looks like the patches brought from upstream:
0017-multipath-get-right-sysfs-value-for-checker_timeout.patch
0018-multipath-handle-offlined-paths.patch
#
# from here
#
0019-multipath-fix-scsi-timeout-code.patch
0020-multipath-make-tgt_node_name-work-for-iscsi-devices.patch
0021-multipath-cleanup-dev_loss_tmo-issues.patch
0022-Fix-for-setting-0-to-fast_io_fail.patch
0023-Fix-fast_io_fail-capping.patch
0024-multipath-enable-getting-uevents-through-libudev.patch
0025-Use-devpath-as-argument-for-sysfs-functions.patch
0026-multipathd-remove-references-to-sysfs_device.patch
0027-multipathd-use-struct-path-as-argument-for-event-pro.patch
0028-Add-global-udev-reference-pointer-to-config.patch
0029-Use-udev-enumeration-during-discovery.patch
0030-use-struct-udev_device-during-discovery.patch
0031-More-debugging-output-when-synchronizing-path-states.patch
0032-Use-struct-udev_device-instead-of-sysdev.patch
0033-discovery-Fixup-cciss-discovery.patch
0035-Use-udev-devices-during-discovery.patch
0036-Remove-all-references-to-hand-craftes-sysfs-code.patch
#
# to here
#
# 0037-multipath-libudev-cleanup-and-bugfixes.patch
# 0038-multipath-check-if-a-device-belongs-to-multipath.patch
# 0039-multipath-and-wwids_file-multipath.conf-option.patch
# 0040-multipath-Check-blacklists-as-soon-as-possible.patch
# 0041-add-wwids-file-cleanup-options.patch
# 0042-add-find_multipaths-option.patch
# 0043-alloc-keywords.patch
# lp1503305_libmultipath_info_on_1st_path_down_dbd131e.patch
In the range 19-36 caused a regression.
Whenever I generate the package (for trusty) including those patches I'm able to
generate a core dump indicating a possible double-free or null-dereference related
to a path removal (that is why I can reproduce with the test case). Unfortunately
it usually explodes inside malloc() or somewhere in glibc.
Using valgrind I was able to verify some free() errors:
==30415== Invalid free() / delete / delete[] / realloc()
==30415== at 0x4C2BDEC: free (vg_replace_malloc.c:473)
==30415== by 0x54E243C: vector_del_slot (vector.c:95)
==30415== by 0x550A516: _remove_map (structs_vec.c:139)
==30415== by 0x550A5C3: _remove_maps (structs_vec.c:170)
==30415== by 0x550A64B: remove_maps (structs_vec.c:181)
==30415== by 0x40713F: configure (main.c:1153)
==30415== by 0x407A74: child (main.c:1419)
==30415== by 0x40837D: main (main.c:1618)
And they are exactly aligned to a core dump (multipathd) I got from another user.
(wrong free was coming from _remove_map). |
We have a problem on multipath-tools.
Usually after a path removal and a re-scan, the multipathd process dies.
I created 2 hosts:
iscsi-server
iscsi-client
With 4 NICs in between them and with a simple multibus multipath. With that I was able to check that there is a regression in multipath-tools.
It looks like the patches brought from upstream:
0017-multipath-get-right-sysfs-value-for-checker_timeout.patch
0018-multipath-handle-offlined-paths.patch
#
# from here
#
0019-multipath-fix-scsi-timeout-code.patch
0020-multipath-make-tgt_node_name-work-for-iscsi-devices.patch
0021-multipath-cleanup-dev_loss_tmo-issues.patch
0022-Fix-for-setting-0-to-fast_io_fail.patch
0023-Fix-fast_io_fail-capping.patch
0024-multipath-enable-getting-uevents-through-libudev.patch
0025-Use-devpath-as-argument-for-sysfs-functions.patch
0026-multipathd-remove-references-to-sysfs_device.patch
0027-multipathd-use-struct-path-as-argument-for-event-pro.patch
0028-Add-global-udev-reference-pointer-to-config.patch
0029-Use-udev-enumeration-during-discovery.patch
0030-use-struct-udev_device-during-discovery.patch
0031-More-debugging-output-when-synchronizing-path-states.patch
0032-Use-struct-udev_device-instead-of-sysdev.patch
0033-discovery-Fixup-cciss-discovery.patch
0035-Use-udev-devices-during-discovery.patch
0036-Remove-all-references-to-hand-craftes-sysfs-code.patch
#
# to here
#
# 0037-multipath-libudev-cleanup-and-bugfixes.patch
# 0038-multipath-check-if-a-device-belongs-to-multipath.patch
# 0039-multipath-and-wwids_file-multipath.conf-option.patch
# 0040-multipath-Check-blacklists-as-soon-as-possible.patch
# 0041-add-wwids-file-cleanup-options.patch
# 0042-add-find_multipaths-option.patch
# 0043-alloc-keywords.patch
# lp1503305_libmultipath_info_on_1st_path_down_dbd131e.patch
In the range 19-36 caused a regression.
Whenever I generate the package (for trusty) including those patches I'm able to generate a core dump indicating a possible double-free or null-dereference related to a path removal (that is why I can reproduce with the test case). Unfortunately it usually explodes inside malloc() or somewhere in glibc.
Using valgrind I was able to verify some free() errors:
==30415== Invalid free() / delete / delete[] / realloc()
==30415== at 0x4C2BDEC: free (vg_replace_malloc.c:473)
==30415== by 0x54E243C: vector_del_slot (vector.c:95)
==30415== by 0x550A516: _remove_map (structs_vec.c:139)
==30415== by 0x550A5C3: _remove_maps (structs_vec.c:170)
==30415== by 0x550A64B: remove_maps (structs_vec.c:181)
==30415== by 0x40713F: configure (main.c:1153)
==30415== by 0x407A74: child (main.c:1419)
==30415== by 0x40837D: main (main.c:1618)
And they are exactly aligned to a core dump (multipathd) I got from another user. (wrong free was coming from _remove_map). |
|
| 2016-01-19 21:41:15 |
Rafael David Tinoco |
multipath-tools (Ubuntu): status |
New |
In Progress |
|
| 2016-01-20 19:23:22 |
Mathieu Trudel-Lapierre |
multipath-tools (Ubuntu): status |
In Progress |
Incomplete |
|
| 2016-01-20 19:23:24 |
Mathieu Trudel-Lapierre |
multipath-tools (Ubuntu): assignee |
|
Mathieu Trudel-Lapierre (mathieu-tl) |
|
| 2016-01-21 01:02:59 |
Mathew Hodson |
multipath-tools (Ubuntu): importance |
Undecided |
High |
|
| 2016-01-21 13:33:27 |
Mathieu Trudel-Lapierre |
attachment added |
|
deal with mpp->alias being allocated wrong https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1535898/+attachment/4554077/+files/multipath-tools_alias_free.debdiff |
|
| 2016-01-21 14:08:42 |
Louis Bouchard |
nominated for series |
|
Ubuntu Precise |
|
| 2016-01-21 14:08:42 |
Louis Bouchard |
bug task added |
|
multipath-tools (Ubuntu Precise) |
|
| 2016-01-21 14:08:49 |
Louis Bouchard |
multipath-tools (Ubuntu Precise): status |
New |
In Progress |
|
| 2016-01-21 14:08:53 |
Louis Bouchard |
multipath-tools (Ubuntu Precise): importance |
Undecided |
High |
|
| 2016-01-21 14:08:58 |
Louis Bouchard |
multipath-tools (Ubuntu Precise): assignee |
|
Louis Bouchard (louis-bouchard) |
|
| 2016-01-21 16:23:01 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2016-02-05 03:41:20 |
Ming-Xun Zhong |
bug |
|
|
added subscriber Ming-Xun Zhong |
| 2016-06-09 07:55:23 |
Louis Bouchard |
marked as duplicate |
|
1532789 |
|
| 2016-06-09 08:01:21 |
Louis Bouchard |
removed duplicate marker |
1532789 |
|
|
| 2016-06-09 15:19:25 |
Louis Bouchard |
nominated for series |
|
Ubuntu Trusty |
|
| 2016-06-09 15:19:25 |
Louis Bouchard |
bug task added |
|
multipath-tools (Ubuntu Trusty) |
|
| 2016-06-09 15:19:33 |
Louis Bouchard |
multipath-tools (Ubuntu Trusty): status |
New |
In Progress |
|
| 2016-06-09 15:19:36 |
Louis Bouchard |
multipath-tools (Ubuntu Trusty): assignee |
|
Louis Bouchard (louis-bouchard) |
|
| 2016-06-09 15:19:44 |
Louis Bouchard |
multipath-tools (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2016-06-29 13:28:12 |
Chris J Arges |
multipath-tools (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2016-06-29 13:28:15 |
Chris J Arges |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-06-29 13:28:17 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2016-06-29 13:28:21 |
Chris J Arges |
tags |
patch |
patch verification-needed |
|
| 2016-07-22 14:46:35 |
Louis Bouchard |
tags |
patch verification-needed |
patch verification-done |
|
| 2016-07-25 15:23:59 |
Dragan S. |
multipath-tools (Ubuntu Precise): assignee |
Louis Bouchard (louis-bouchard) |
Dragan S. (dragan-s) |
|
| 2016-07-25 15:24:03 |
Dragan S. |
multipath-tools (Ubuntu Trusty): assignee |
Louis Bouchard (louis-bouchard) |
Dragan S. (dragan-s) |
|
| 2016-07-25 15:24:15 |
Dragan S. |
multipath-tools (Ubuntu): assignee |
Mathieu Trudel-Lapierre (cyphermox) |
Dragan S. (dragan-s) |
|
| 2016-07-26 06:46:59 |
Steve Langasek |
multipath-tools (Ubuntu Precise): assignee |
Dragan S. (dragan-s) |
Louis Bouchard (louis-bouchard) |
|
| 2016-07-26 06:47:52 |
Steve Langasek |
multipath-tools (Ubuntu Trusty): assignee |
Dragan S. (dragan-s) |
Louis Bouchard (louis-bouchard) |
|
| 2016-07-26 06:47:58 |
Steve Langasek |
multipath-tools (Ubuntu): assignee |
Dragan S. (dragan-s) |
Mathieu Trudel-Lapierre (cyphermox) |
|
| 2016-07-26 13:46:48 |
Martin Pitt |
tags |
patch verification-done |
patch verification-neededd |
|
| 2016-07-26 13:46:51 |
Martin Pitt |
tags |
patch verification-neededd |
patch verification-needed |
|
| 2016-09-09 10:56:23 |
Louis Bouchard |
description |
We have a problem on multipath-tools.
Usually after a path removal and a re-scan, the multipathd process dies.
I created 2 hosts:
iscsi-server
iscsi-client
With 4 NICs in between them and with a simple multibus multipath. With that I was able to check that there is a regression in multipath-tools.
It looks like the patches brought from upstream:
0017-multipath-get-right-sysfs-value-for-checker_timeout.patch
0018-multipath-handle-offlined-paths.patch
#
# from here
#
0019-multipath-fix-scsi-timeout-code.patch
0020-multipath-make-tgt_node_name-work-for-iscsi-devices.patch
0021-multipath-cleanup-dev_loss_tmo-issues.patch
0022-Fix-for-setting-0-to-fast_io_fail.patch
0023-Fix-fast_io_fail-capping.patch
0024-multipath-enable-getting-uevents-through-libudev.patch
0025-Use-devpath-as-argument-for-sysfs-functions.patch
0026-multipathd-remove-references-to-sysfs_device.patch
0027-multipathd-use-struct-path-as-argument-for-event-pro.patch
0028-Add-global-udev-reference-pointer-to-config.patch
0029-Use-udev-enumeration-during-discovery.patch
0030-use-struct-udev_device-during-discovery.patch
0031-More-debugging-output-when-synchronizing-path-states.patch
0032-Use-struct-udev_device-instead-of-sysdev.patch
0033-discovery-Fixup-cciss-discovery.patch
0035-Use-udev-devices-during-discovery.patch
0036-Remove-all-references-to-hand-craftes-sysfs-code.patch
#
# to here
#
# 0037-multipath-libudev-cleanup-and-bugfixes.patch
# 0038-multipath-check-if-a-device-belongs-to-multipath.patch
# 0039-multipath-and-wwids_file-multipath.conf-option.patch
# 0040-multipath-Check-blacklists-as-soon-as-possible.patch
# 0041-add-wwids-file-cleanup-options.patch
# 0042-add-find_multipaths-option.patch
# 0043-alloc-keywords.patch
# lp1503305_libmultipath_info_on_1st_path_down_dbd131e.patch
In the range 19-36 caused a regression.
Whenever I generate the package (for trusty) including those patches I'm able to generate a core dump indicating a possible double-free or null-dereference related to a path removal (that is why I can reproduce with the test case). Unfortunately it usually explodes inside malloc() or somewhere in glibc.
Using valgrind I was able to verify some free() errors:
==30415== Invalid free() / delete / delete[] / realloc()
==30415== at 0x4C2BDEC: free (vg_replace_malloc.c:473)
==30415== by 0x54E243C: vector_del_slot (vector.c:95)
==30415== by 0x550A516: _remove_map (structs_vec.c:139)
==30415== by 0x550A5C3: _remove_maps (structs_vec.c:170)
==30415== by 0x550A64B: remove_maps (structs_vec.c:181)
==30415== by 0x40713F: configure (main.c:1153)
==30415== by 0x407A74: child (main.c:1419)
==30415== by 0x40837D: main (main.c:1618)
And they are exactly aligned to a core dump (multipathd) I got from another user. (wrong free was coming from _remove_map). |
[SRU justification]
Without this patch, multipathd may exit in SEGV in trying to add a map that aleady exists
[Impact]
multipathd crashes with SIGSEGV
A typical trace of such a situation is a message similar to this one in /var/log/syslog :
multipathd: 360060160164034004cd59cfdb22ce611: failed in domap for addition of new path sdr
[Fix]
Check if the map already exists and do a RELOAD in domap() instead of failing.
[Test Case]
Problem was encountered in a complex Openstack test environment where the following was done :
A test tool which runs which :
- first boots a number of virtual machines.
- then it creates a number of threads and in each thread it
creates volumes, takes snapshots of the volumes, and attaches the volumes to the initially booted virtual machines. After a short while the volumes are detached, and snapshots and volumes are deleted.
Running this tool overnight normally result in running in the multipathd SEGV situation.
[Regression]
This is a straight backport of the code being used in 0.5.0. No regression is to be expected.
It is important to note that the reproducer in the original description did not lead to such a problem.
[Original description of the problem]
We have a problem on multipath-tools.
Usually after a path removal and a re-scan, the multipathd process dies.
I created 2 hosts:
iscsi-server
iscsi-client
With 4 NICs in between them and with a simple multibus multipath. With that I was able to check that there is a regression in multipath-tools.
It looks like the patches brought from upstream:
0017-multipath-get-right-sysfs-value-for-checker_timeout.patch
0018-multipath-handle-offlined-paths.patch
#
# from here
#
0019-multipath-fix-scsi-timeout-code.patch
0020-multipath-make-tgt_node_name-work-for-iscsi-devices.patch
0021-multipath-cleanup-dev_loss_tmo-issues.patch
0022-Fix-for-setting-0-to-fast_io_fail.patch
0023-Fix-fast_io_fail-capping.patch
0024-multipath-enable-getting-uevents-through-libudev.patch
0025-Use-devpath-as-argument-for-sysfs-functions.patch
0026-multipathd-remove-references-to-sysfs_device.patch
0027-multipathd-use-struct-path-as-argument-for-event-pro.patch
0028-Add-global-udev-reference-pointer-to-config.patch
0029-Use-udev-enumeration-during-discovery.patch
0030-use-struct-udev_device-during-discovery.patch
0031-More-debugging-output-when-synchronizing-path-states.patch
0032-Use-struct-udev_device-instead-of-sysdev.patch
0033-discovery-Fixup-cciss-discovery.patch
0035-Use-udev-devices-during-discovery.patch
0036-Remove-all-references-to-hand-craftes-sysfs-code.patch
#
# to here
#
# 0037-multipath-libudev-cleanup-and-bugfixes.patch
# 0038-multipath-check-if-a-device-belongs-to-multipath.patch
# 0039-multipath-and-wwids_file-multipath.conf-option.patch
# 0040-multipath-Check-blacklists-as-soon-as-possible.patch
# 0041-add-wwids-file-cleanup-options.patch
# 0042-add-find_multipaths-option.patch
# 0043-alloc-keywords.patch
# lp1503305_libmultipath_info_on_1st_path_down_dbd131e.patch
In the range 19-36 caused a regression.
Whenever I generate the package (for trusty) including those patches I'm able to generate a core dump indicating a possible double-free or null-dereference related to a path removal (that is why I can reproduce with the test case). Unfortunately it usually explodes inside malloc() or somewhere in glibc.
Using valgrind I was able to verify some free() errors:
==30415== Invalid free() / delete / delete[] / realloc()
==30415== at 0x4C2BDEC: free (vg_replace_malloc.c:473)
==30415== by 0x54E243C: vector_del_slot (vector.c:95)
==30415== by 0x550A516: _remove_map (structs_vec.c:139)
==30415== by 0x550A5C3: _remove_maps (structs_vec.c:170)
==30415== by 0x550A64B: remove_maps (structs_vec.c:181)
==30415== by 0x40713F: configure (main.c:1153)
==30415== by 0x407A74: child (main.c:1419)
==30415== by 0x40837D: main (main.c:1618)
And they are exactly aligned to a core dump (multipathd) I got from another user. (wrong free was coming from _remove_map). |
|
| 2016-09-09 16:37:33 |
Steve Langasek |
tags |
patch verification-needed |
patch verification-done |
|
| 2016-09-12 13:48:40 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Server Team |
| 2016-09-12 21:09:53 |
Martin Pitt |
tags |
patch verification-done |
patch verification-needed |
|
| 2016-09-28 08:19:42 |
Louis Bouchard |
tags |
patch verification-needed |
patch verification-done |
|
| 2016-09-28 14:53:25 |
Launchpad Janitor |
multipath-tools (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2016-09-28 14:53:32 |
Chris J Arges |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-06-07 20:28:14 |
Nish Aravamudan |
multipath-tools (Ubuntu Precise): status |
In Progress |
Won't Fix |
|
| 2017-06-07 20:28:21 |
Nish Aravamudan |
removed subscriber Ubuntu Server Team |
|
|
|
| 2019-05-18 17:20:19 |
Rafael David Tinoco |
removed subscriber Rafael David Tinoco |
|
|
|
