Activity log for bug #1616213

Date Who What changed Old value New value Message
2016-08-23 20:51:16 Dragan S. bug added bug
2016-08-23 20:51:30 Dragan S. multipath-tools (Ubuntu): assignee Dragan S. (dragan-s)
2016-08-23 20:51:53 Dragan S. multipath-tools (Ubuntu): status New In Progress
2016-09-09 14:52:58 Louis Bouchard nominated for series Ubuntu Trusty
2016-09-09 14:52:58 Louis Bouchard bug task added multipath-tools (Ubuntu Trusty)
2016-09-12 08:41:31 Louis Bouchard multipath-tools (Ubuntu Trusty): status New Incomplete
2016-09-12 08:41:37 Louis Bouchard multipath-tools (Ubuntu Trusty): status Incomplete In Progress
2016-09-12 08:41:48 Louis Bouchard multipath-tools (Ubuntu Trusty): assignee Dragan S. (dragan-s)
2016-09-12 08:42:23 Louis Bouchard multipath-tools (Ubuntu): importance Undecided Medium
2016-09-12 08:42:26 Louis Bouchard multipath-tools (Ubuntu Trusty): importance Undecided Medium
2016-10-01 00:08:40 Dragan S. attachment added Fix debdiff https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+attachment/4752077/+files/fix-lp1616213.debdiff
2016-10-01 00:10:04 Dragan S. bug added subscriber Ubuntu Sponsors Team
2016-10-02 01:09:52 Mathew Hodson tags sts trusty patch sts trusty
2016-10-06 18:40:07 Dragan S. description On ubuntu trusty 14.04.4 in multipath-tools version 0.4.9-3ubuntu7.14 there is bug in multipathd on shutdown. The code will access pathvec pointer which is a valid address: Reading symbols from /sbin/multipathd...Reading symbols from /usr/lib/debug//sbin/multipathd...done. done. [New LWP 41631] [New LWP 41584] [New LWP 41633] [New LWP 41632] [New LWP 41582] [New LWP 41583] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/sbin/multipathd'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004075db in checkerloop (ap=0x1b81040) at main.c:1150 1150 vector_foreach_slot (vecs->pathvec, pp, i) { (gdb) list 1145 pthread_cleanup_push(cleanup_lock, &vecs->lock); 1146 lock(vecs->lock); 1147 condlog(4, "tick"); 1148 1149 if (vecs->pathvec) { 1150 vector_foreach_slot (vecs->pathvec, pp, i) { 1151 check_path(vecs, pp); 1152 } 1153 } 1154 if (vecs->mpvec) { Pathvec is a valid pointer: (gdb) p vecs->pathvec $1 = (vector) 0x1b81280 But the contents of the structure are just garbage: (gdb) p *vecs->pathvec $2 = {allocated = 1651076143, slot = 0x756e696c2d34365f} (gdb) [Impact] * During "service multipath-tools stop" multipath daemon is trying to cleanup and shut down several concurrent threads. At times depending on a race condition between two threads, one thread might free resources that are still used by another thread. This is causing the multipathd to dump crash core on stop events. * Fix should be backported to trusty to avoid more support issues being filed. * This change delays freeing resources that another thread is still using. [Test Case] * install multipath-tools, create a basic multipath.conf with devices under management. Run: "service multipath-tools start" run I/O on devices and keep the system CPU busy, then run "service multipath-tools stop". [Regression Potential] * There should be no regression potential with this change, this problem happens on the exit path and we are only delaying a free call. [Original Description] On ubuntu trusty 14.04.4 in multipath-tools version 0.4.9-3ubuntu7.14 there is bug in multipathd on shutdown. The code will access pathvec pointer which is a valid address: Reading symbols from /sbin/multipathd...Reading symbols from /usr/lib/debug//sbin/multipathd...done. done. [New LWP 41631] [New LWP 41584] [New LWP 41633] [New LWP 41632] [New LWP 41582] [New LWP 41583] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/sbin/multipathd'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004075db in checkerloop (ap=0x1b81040) at main.c:1150 1150 vector_foreach_slot (vecs->pathvec, pp, i) { (gdb) list 1145 pthread_cleanup_push(cleanup_lock, &vecs->lock); 1146 lock(vecs->lock); 1147 condlog(4, "tick"); 1148 1149 if (vecs->pathvec) { 1150 vector_foreach_slot (vecs->pathvec, pp, i) { 1151 check_path(vecs, pp); 1152 } 1153 } 1154 if (vecs->mpvec) { Pathvec is a valid pointer: (gdb) p vecs->pathvec $1 = (vector) 0x1b81280 But the contents of the structure are just garbage: (gdb) p *vecs->pathvec $2 = {allocated = 1651076143, slot = 0x756e696c2d34365f} (gdb)
2016-10-14 09:49:14 Louis Bouchard multipath-tools (Ubuntu): status In Progress Invalid
2016-10-14 17:21:35 Brian Murray multipath-tools (Ubuntu Trusty): status In Progress Fix Committed
2016-10-14 17:21:38 Brian Murray bug added subscriber Ubuntu Stable Release Updates Team
2016-10-14 17:21:45 Brian Murray bug added subscriber SRU Verification
2016-10-14 17:21:49 Brian Murray tags patch sts trusty patch sts trusty verification-needed
2016-10-14 17:22:04 Brian Murray removed subscriber Ubuntu Sponsors Team
2016-10-18 22:32:35 Dragan S. tags patch sts trusty verification-needed patch sts trusty verification-done
2016-11-02 12:47:17 Launchpad Janitor multipath-tools (Ubuntu Trusty): status Fix Committed Fix Released
2016-11-02 12:47:26 Chris J Arges removed subscriber Ubuntu Stable Release Updates Team