Xenial 32-bit: multipath exits with SIGSEGV

[Amit Oren]

[Impact]

 * The CLI for multipath segfaults when on 32-bit platforms.

 * As example, a simple 'sudo multipathd -k"show paths"' will blow up,
   assuming you have a multipath configured.

 * The ultimate cause of the bug is the codebase not correctly dealing with bit flags that have exceeded the 32-bit natural word size on 32-bit architectures. The fix has been cherrypicked from upstream.

[Test Case]

 * You'll need to setup a 32-bit (e.g. i386) multipath environment to test. I used qemu/kvm.

 * See https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1611360/comments/4

 * Once configured, type "sudo multipathd -k"show paths" and it will
fail with a segfault on 32-bit architectures, but succeed on 64-bit architectures.

[Regression Potential]

 * Limited potential as only affects flag bits > 32-bits. In addition to testing commands affected by these bits on 32-bit platform (i386), I've regression tested that these commands work on 64-bit platform (amd64).

[Amit Oren (amito)]

Ping?

[Jon Grimm (jgrimm)]

Looks like this is fixed upstream by:
http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commitdiff;h=62acabf32795076460e395cb06417cd53b7e1fe1

,already in debian too, so we'll pick it up with next merge, but otherwise looks doable. Just parking my thoughts. Looking at the change, will be pretty clean or minimal massaging to SRU back to x, t as needed.

[Jon Grimm (jgrimm)]

OK, I've verified I can reproduce the segfault with a 32-bit i386 xenial kvm guest. Much futzing around to get the scsi multipathing working, I'll doc that when I get a chance so I can reproduce, but wanted to document the test results and make my fix available via PPA for others.

Before fix:

ubuntu@x32multipath:~$ sudo multipathd -k
multipathd> show paths
ubuntu@x32multipath:~$

But, dmesg shows the segfault:

[ 142.603813] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5
[ 142.605344] sd 2:0:0:0: Attached scsi generic sg0 type 0
[ 142.605744] sd 2:0:0:0: [sda] 40960 512-byte logical blocks: (21.0 MB/20.0 MiB)
[ 142.606901] sd 2:0:0:0: [sda] Write Protect is off
[ 142.606908] sd 2:0:0:0: [sda] Mode Sense: 63 00 00 08
[ 142.607054] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 142.610992] sda: sda1
[ 142.611780] sd 2:0:0:0: [sda] Attached SCSI disk
[ 151.914198] scsi 2:0:0:1: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5
[ 151.917004] sd 2:0:0:1: Attached scsi generic sg1 type 0
[ 151.917385] sd 2:0:0:1: [sdb] 40960 512-byte logical blocks: (21.0 MB/20.0 MiB)
[ 151.917755] sd 2:0:0:1: [sdb] Write Protect is off
[ 151.917764] sd 2:0:0:1: [sdb] Mode Sense: 63 00 00 08
[ 151.917864] sd 2:0:0:1: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 151.920866] sdb: sdb1
[ 151.921576] sd 2:0:0:1: [sdb] Attached SCSI disk
[ 151.975052] device-mapper: multipath round-robin: version 1.0.0 loaded
[ 213.768916] show_signal_msg: 12 callbacks suppressed
[ 213.768929] multipathd[670]: segfault at 0 ip b761b7cd sp b71c6040 error 4 in libmultipath.so.0[b75f9000+45000]

^^^^ blows up as per submitter ^^^

After update with fix:

ubuntu@x32multipath:~$ sudo multipathd -k
multipathd> show paths
hcil dev dev_t pri dm_st chk_st dev_st next_check
2:0:0:0 sda 8:0 1 active ready running XXXX...... 4/10
2:0:0:1 sdb 8:16 1 active ready running XXXXX..... 5/10
multipathd>

So, looking good. Will get back to this next week.

If anyone else seeing this problem is interested in testing the fix:

Available via PPA (has yakkety and xenial, but I've only tested xenial):

sudo add-apt-repository ppa:jgrimm/multipath-1611360
sudo apt-get update
sudo apt-get upgrade

cheers,

[Jon Grimm (jgrimm)]

Here were the steps I used to recreate this bug.

1) Create 32-bit ubuntu guest (xenial or yakkety). Note: my guest is named 'x32multipath'.

$ uvt-kvm create x32multipath release=xenial arch=i386

2) In your guest, install multipath tools

$ sudo apt install multipath-tools

3) From host, attach virtio-scsi controller into guest
(Note, you may have to disable apparmor - I ran into what was possibly a bug blocking me, but it doesn't seem to reproduce now that I've re-installed apparmor).

$ sudo echo "<controller type='scsi' model='virtio-scsi'/>" > ./scsi.xml
$ sudo virsh attach-device --config ./scsi.xml
$ sudo virsh shutdown x32multipath
$ sudo virsh start x32multipath

4) Create a disk-image for attaching to the guest.

$ sudo qemu-img create /tmp/scsidisk.img 20M
Formatting '/tmp/scsidisk.img', fmt=raw size=20971520

5) Attach this disk to the guest as both sda and sdb.

$ sudo virsh attach-disk x32multipath /tmp/scsidisk.img sda --cache none --serial 0001
$ sudo virsh attach-disk x32multipath /tmp/scsidisk.img sdb --cache none --serial 0002

6) From your guest, execute show paths.

$ sudo multipathd -k"show paths"
error -5 receiving packet

You'll find a segfault in your 'dmesg':
...
...
[12979.571708] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5
[12979.574653] sd 2:0:0:0: Attached scsi generic sg0 type 0
[12979.574889] sd 2:0:0:0: [sda] 40960 512-byte logical blocks: (21.0 MB/20.0 MiB)
[12979.575290] sd 2:0:0:0: [sda] Write Protect is off
[12979.575294] sd 2:0:0:0: [sda] Mode Sense: 63 00 00 08
[12979.575418] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[12979.578059] sd 2:0:0:0: [sda] Attached SCSI disk
[12985.008262] scsi 2:0:0:1: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5
[12985.010632] sd 2:0:0:1: Attached scsi generic sg1 type 0
[12985.010793] sd 2:0:0:1: [sdb] 40960 512-byte logical blocks: (21.0 MB/20.0 MiB)
[12985.011144] sd 2:0:0:1: [sdb] Write Protect is off
[12985.011148] sd 2:0:0:1: [sdb] Mode Sense: 63 00 00 08
[12985.011261] sd 2:0:0:1: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[12985.013457] sd 2:0:0:1: [sdb] Attached SCSI disk
[13071.000211] multipathd[2387]: segfault at 0 ip b75d492a sp b7149030 error 4 in libmultipath.so.0[b75b2000+45000]

AFTER FIX:

After applying the update you should get something more pleasant:
$ sudo multipathd -k"show paths"
hcil dev dev_t pri dm_st chk_st dev_st next_check
2:0:0:0 sda 8:0 1 active ready running XXX....... 3/10
2:0:0:1 sdb 8:16 1 active ready running XXXX...... 4/10

[Jon Grimm (jgrimm)]

Debdiff for sponsorship once zesty is open.

[Jon Grimm (jgrimm)]

Added DEP3 headers

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.5.0+git1.656f8865-5ubuntu8

---------------
multipath-tools (0.5.0+git1.656f8865-5ubuntu8) zesty; urgency=medium

   * Cherrypick from upstream:
     - Use 64-bit int for command key (LP: #1611360)

 -- Jon Grimm <email address hidden> Fri, 14 Oct 2016 15:49:14 -0500

[Amit Oren (amito)]

Thanks Jon Grimm and everyone else, but can you also fix it in Xenial?
We are using Xenial on our servers as it is LTS, and giving support for Xenial to our customers for this very reason, and we did not intend to upgrade to Zesty anytime soon. I don't want to use pinning for that specific package or any sort of bug-prone solution...

[Jon Grimm (jgrimm)]

Hi there Amit!

Yes, I will fix it in xenial & yakkety next. The Ubuntu development
process requires that it be fixed & vetted in the development release (in
this case 'zesty') first, working on the SRU (Stable Release Update) now.
:)

On Sat, Oct 22, 2016 at 4:10 AM, Amit Oren <email address hidden> wrote:

> Thanks Jon Grimm and everyone else, but can you also fix it in Xenial?
> We are using Xenial on our servers as it is LTS, and giving support for
> Xenial to our customers for this very reason, and we did not intend to
> upgrade to Zesty anytime soon. I don't want to use pinning for that
> specific package or any sort of bug-prone solution...
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1611360
>
> Title:
> Xenial 32-bit: multipath exits with SIGSEGV
>
> Status in multipath-tools package in Ubuntu:
> Fix Released
>
> Bug description:
> Ubuntu release:
> Description: Ubuntu 16.04 LTS
> Release: 16.04
>
> Package version: 0.5.0+git1.656f8865-5ubuntu2.1
>
> The current version of multipath-tools for Xenial
> (0.5.0+git1.656f8865-5ubuntu2.1) causes segmentation fault when trying to
> run, for example, "show paths" from the daemon.
> When further investigating the issue, it seems that since Ubuntu 14.04,
> the enum describing the keyword type in cli.h is now of size 35. This leads
> to overflows as later on values are left-shifted by the values in these
> enum, and some of those exceed 31 (the maximum legal shift amount in 32-bit
> int).
> This leads to wrong choice of the handler in the command parsing stage
> (cli.c - find_handler). Specifically in "show paths" it leads to choosing
> cli_list_paths_fmt instead of cli_list_paths, and later to null
> dereferencing and segmentation fault.
>
> GDB Backtrace:
> #0 snprint_path_header (line=0xb78010e0 "", len=1200, format=0x0) at
> print.c:795
> #1 0x08052a6e in show_paths (r=0xb7b52210, len=0xb7b52208,
> vecs=0x809f650, style=0x0, pretty=1) at cli_handlers.c:47
> #2 0x080532af in cli_list_paths_fmt (v=0xb7800678, reply=0xb7b52210,
> len=0xb7b52208, data=0x809f650) at cli_handlers.c:235
> #3 0x08052369 in parse_cmd (cmd=0xb7800f68 "show paths",
> reply=0xb7b52210, len=0xb7b52208, data=0x809f650) at cli.c:438
> #4 0x0804cf49 in uxsock_trigger (str=0xb7800f68 "show paths",
> reply=0xb7b52210, len=0xb7b52208, trigger_data=0x809f650) at main.c:780
> #5 0x08050e5a in uxsock_listen (uxsock_trigger=0x804cf00
> <uxsock_trigger>, trigger_data=0x809f650) at uxlsnr.c:252
> #6 0x0804da3b in uxlsnrloop (ap=0x809f650) at main.c:946
> #7 0xb7fb82b5 in start_thread (arg=0xb7b52b40) at pthread_create.c:333
> #8 0xb7da216e in clone () at ../sysdeps/unix/sysv/linux/
> i386/clone.S:114
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1611360/+
> subscriptions
>

--
Jon Grimm
Engineering Manager, Ubuntu Server
Canonical Ltd

[Amit Oren (amito)]

Hi Jon!

Thanks, I was not aware of that :)

[Jon Grimm (jgrimm)]

Yakkety debdiff

[Jon Grimm (jgrimm)]

Xenial debdiff

[Jon Grimm (jgrimm)]

Subscribed ubuntu-sponsors for upload assist for x & y releases.

[Louis Bouchard (louis)]

Unsubscribing ubuntu-sru, both X & Y debdiffs sponsored

[Martin Pitt (pitti)]

Hello Amit, or anyone else affected,

Accepted multipath-tools into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.5.0+git1.656f8865-5ubuntu7.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Amit Oren (amito)]

Thanks Martin. As previously posted, I will test this package as soon as it is released to the Xenial repo.

[Martin Pitt (pitti)]

Hello Amit, or anyone else affected,

Accepted multipath-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.5.0+git1.656f8865-5ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Jon Grimm (jgrimm)]

Verified my test case on yakkety. I'll wait on marking verification-done for Amit on xenial, as he has real hardware, and reported the original issue.

[Jon Grimm (jgrimm)]

Tested package multipath-tools | 0.5.0+git1.656f8865-5ubuntu7.1 on yakkety on i386 architecture with test case described in comment https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1611360/comments/4

[Amit Oren (amito)]

Tested package multipath-tools | 0.5.0+git1.656f8865-5ubuntu7.1 successfully on xenial on i386 architecture. The test case basically follows the lines of the one described by Jon Grimm in comment #19.

[Martin Pitt (pitti)]

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.5.0+git1.656f8865-5ubuntu2.3

---------------
multipath-tools (0.5.0+git1.656f8865-5ubuntu2.3) xenial; urgency=medium

  * Cherrypick from upstream:
    - Use 64-bit int for command key (LP: #1611360)

 -- Jon Grimm <email address hidden> Mon, 07 Nov 2016 17:04:42 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.5.0+git1.656f8865-5ubuntu7.1

---------------
multipath-tools (0.5.0+git1.656f8865-5ubuntu7.1) yakkety; urgency=medium

  * Cherrypick from upstream:
    - Use 64-bit int for command key (LP: #1611360)

 -- Jon Grimm <email address hidden> Mon, 07 Nov 2016 17:06:22 +0100