multipath segmentation Fault (libmultipath: update waiter handling)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
multipath-tools (Debian) |
Fix Released
|
Unknown
|
|||
multipath-tools (Ubuntu) |
Fix Released
|
Undecided
|
Rafael David Tinoco | ||
Precise |
Fix Released
|
Undecided
|
Rafael David Tinoco | ||
Trusty |
Fix Released
|
Undecided
|
Rafael David Tinoco |
Bug Description
[Impact]
* Multipath can cause segmentation fault due to wrong code and can
possibly cause user to loose access to multipath devices.
[Test Case]
* To use multipath and wait for the problem to occur sometime (inevitable).
[Regression Potential]
* Patch 1/4 tries to fix the issue. Patch 2/4 fixes the 1/4.
* Patch 3/4 discovers 1/4 was no good. Patch 4/4 fixes 3/4.
* Fix based on upstream code (96f8146) + subsequent patches.
* Followed this code development until the issue was addressed.
[Other Info]
* Original bug description:
----------------
It was brought to me (~inaddy) the following situation with multipathd:
#####
Program terminated with signal 6, Aborted.
#0 0x00007fbc6ae09445 in raise () from /lib/x86_
libc.so.6
(gdb) bt
#0 0x00007fbc6ae09445 in raise () from /lib/x86_
libc.so.6
#1 0x00007fbc6ae0cbab in abort () from /lib/x86_
libc.so.6
#2 0x00007fbc6ae0210e in ?? () from /lib/x86_
libc.so.6
#3 0x00007fbc6ae021b2 in __assert_fail () from /lib/x86_
libc.so.6
#4 0x00007fbc6b849efb in pthread_mutex_lock () from /lib/x86_
libpthread.so.0
#5 0x00007fbc6b1cba5f in free_waiter (data=0x1691de0) at waiter.c:44
#6 0x00007fbc6b1cc25a in waitevent (et=0x1691de0) at waiter.c:204
#7 0x00007fbc6b847e9a in start_thread () from /lib/x86_
libpthread.so.0
#8 0x00007fbc6aec54bd in clone () from /lib/x86_
libc.so.6
#9 0x0000000000000000 in ?? ()
-------
#5 0x00007fbc6b1cba5f in free_waiter (data=0x1691de0) at waiter.c:44
44 lock(wp>
vecs>
lock);
(gdb) print wp>
vecs>
lock
$1 = {mutex = 0x168c280, depth = 1}
In pthread_
#4 0x00007fbc6b849efb in __pthread_
62 assert (mutex>_
data._owner == 0);
In this run:
(gdb) p *wp>
vecs>
lock>
mutex
$3 = {_data = {lock = 1, __count = 0, __owner = 49, __nusers = 0, __kind = 0, __spins = 0, __list = {_prev = 0x0, __next = 0xffffffff}},
__size = "\001\000\
so __owner is 49 and not 0.
Note that 49 is somewhat strange; it's expected to be a pid_t obtained via
pid_t id = THREAD_GETMEM (THREAD_SELF, tid);
According to https:/
The multipath-tools package is up to date (0.4.9-3ubuntu5)
I do not find obvious thing related in http://
http://
#####
In between Precise's version and Upstream there are the following patches touching waiter.c:
d887f4a = signal waiter thread to stop waiting on dm events
5ee9f71 = simplify multipath signal handlers
af4fd6d = Fix race condition in stop_waiter_
e1fcc59 = multipath: clean up code for stopping the waiter threads
03ec4ef = multipath: fix shutdown crashes
4dfdaf2 = multipath: Update multipath device on show topology
c301a3f = Race condition when calling stop_waiter_
96f8146 = libmultipath: update waiter handling
This specific one: 96f8146 (libmultipath: update waiter handling)
"""
The current 'waiter' structure accesses fields which belong
to the main 'mpp' structure, which has a totally different
lifetime.
"""
Shows that due to different lifetime between different structures, there can be use-after-free segfaults (what seems to be happening).
waiter.c:44 = lock(wp-
Changed in multipath-tools (Ubuntu): | |
assignee: | nobody → Rafael David Tinoco (inaddy) |
status: | New → Confirmed |
description: | updated |
description: | updated |
description: | updated |
Changed in multipath-tools (Ubuntu Trusty): | |
status: | New → Confirmed |
Changed in multipath-tools (Ubuntu Precise): | |
assignee: | nobody → Rafael David Tinoco (inaddy) |
status: | New → Confirmed |
Changed in multipath-tools (Ubuntu Trusty): | |
assignee: | nobody → Rafael David Tinoco (inaddy) |
Changed in multipath-tools (Debian): | |
importance: | Undecided → Unknown |
status: | New → Unknown |
Changed in multipath-tools (Debian): | |
status: | Unknown → Fix Released |
tags: | added: cts |
Attaching SRU proposal.
"""
Description: [PATCH] libmultipath: update waiter handling
The current 'waiter' structure accesses fields which belong
to the main 'mpp' structure, which has a totally different
lifetime. With this patch most of these dependencies are
removed and the 'waiter' structure can run independently
of the main 'mpp' structure, reducing the risk of
use-after-free faults.
"""
Judging by upstream fix:
# <email address hidden> :/bugs/ 00067428/ sources/ upstream$ git tag --contains 96f8146
0.5.0
All Ubuntu versions are affected:
# <email address hidden> :/bugs/ 00067428/ sources/ upstream$ rmadison multipath-tools .10.04. 2 | lucid-updates | source, amd64, armel, i386, ia64, powerpc, sparc
multipath-tools | 0.4.8-14ubuntu4 | lucid | source, amd64, armel, i386, ia64, powerpc, sparc
multipath-tools | 0.4.8-14ubuntu4
multipath-tools | 0.4.9-3ubuntu5 | precise | source, amd64, armel, armhf, i386, powerpc
multipath-tools | 0.4.9-3ubuntu5.1 | precise-updates | source, amd64, armel, armhf, i386, powerpc
multipath-tools | 0.4.9-3ubuntu7 | trusty | source, amd64, arm64, armhf, i386, powerpc, ppc64el
multipath-tools | 0.4.9-3ubuntu8 | utopic | source, amd64, arm64, armhf, i386, powerpc, ppc64el