multipath -ll is not showing the disks which are actually multipath
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| The Ubuntu-power-systems project |
Fix Released
|
Critical
|
Canonical Kernel Team | ||
| linux (Ubuntu) |
Fix Released
|
Critical
|
Ubuntu on IBM Power Systems Bug Triage | ||
| Xenial |
Fix Released
|
Critical
|
Unassigned | ||
| Zesty |
Fix Released
|
Critical
|
Canonical Kernel Team | ||
| Artful |
Fix Released
|
Critical
|
Ubuntu on IBM Power Systems Bug Triage | ||
Bug Description
[Impact]
* The number of available AIO contexts is severely limited
on systems with a large number of possible CPUs
(e.g., IBM POWER8 processors w/ 20ish cores * 8 threads/core,
and other multithreaded server-class processors).
* This prevents application such as multipath/directio checker
to provide all of the available devices to the system.
* Other applications which depend on AIO can be affected/limited.
* The patch fixes how aio increments the number of active contexts
(seen in /proc/sys/
limit (seen in /proc/sys/
[Test Case]
* A synthetic test-case is attached (io_setup_v2.c) and demonstrated
(original/
* Trying to perform multipath discovery in debug/verbose mode
(i.e., "multipath -v3" command) with sufficient number of
individual paths using the "directio" path checker should
demonstrate the problem/solution as well (i.e., presence or
not of "io_setup failed" messages).
[Regression Potential]
* Note the fix is trivial and has been tested by several users,
even caused the introduction of a new test-case in "libaio";
(but that can never be a strong enough reason for no more errors).
* Applications which use aio with small "nr_events" value as argument
to "io_setup()" now have access to a much larger number of aio contexts;
but hopefully those apps are already only requesting what they need,
not trying to get more and more.
* Applications which relied in the _incorrect_ behavior of '/proc/
being possibly greater than '/proc/
but those apps should be fixed.
Problem Description
=======
I am facing this issue for Texan Flash storage 840 disks which are coming from coho and salfish adapter
coho adapter with 840 storage is 3G disks and salfish adapter with 840 is 12G disks
I am able to see those disks in lsblk o/p but not in multipath -ll comamnd
0004:01:00.0 Coho: Saturn-X U78C9.001.
0004:01:00.1 Coho: Saturn-X U78C9.001.
0005:09:00.0 Sailfish: QLogic 8GB U78C9.001.
0005:09:00.1 Sailfish: QLogic 8GB U78C9.001.
root@luckyv1:
root@luckyv1:
root@luckyv1:
== Comment: #3 - Luciano Chavez <email address hidden> - 2016-09-20 20:22:20 ==
I edited /etc/multipath.conf and added
verbosity 6
to crank up the output and ran multipath -ll and saved it off to a text file (attached). All the using the directio checker failed and those using the tur checker seem to work.
Sep 20 20:07:36 | loading //lib/multipath
Sep 20 20:07:36 | loading //lib/multipath
Sep 20 20:07:36 | Discover device /sys/devices/
Sep 20 20:07:36 | sdai: udev property ID_WWN whitelisted
Sep 20 20:07:36 | sdai: not found in pathvec
Sep 20 20:07:36 | sdai: mask = 0x25
Sep 20 20:07:36 | sdai: dev_t = 66:32
Sep 20 20:07:36 | open '/sys/devices/
Sep 20 20:07:36 | sdai: size = 20971520
Sep 20 20:07:36 | sdai: vendor = IBM
Sep 20 20:07:36 | sdai: product = FlashSystem-9840
Sep 20 20:07:36 | sdai: rev = 1442
Sep 20 20:07:36 | sdai: h:b:t:l = 3:0:0:0
Sep 20 20:07:36 | SCSI target 3:0:0 -> FC rport 3:0-2
Sep 20 20:07:36 | sdai: tgt_node_name = 0x500507605e839800
Sep 20 20:07:36 | open '/sys/devices/
Sep 20 20:07:36 | sdai: path state = running
Sep 20 20:07:36 | sdai: get_state
Sep 20 20:07:36 | sdai: path_checker = directio (internal default)
Sep 20 20:07:36 | sdai: checker timeout = 30 ms (internal default)
Sep 20 20:07:36 | io_setup failed
Sep 20 20:07:36 | sdai: checker init failed
== Comment: #7 - Mauricio Faria De Oliveira <email address hidden> - 2016-09-27 18:32:57 ==
The function is failing at the io_setup() system call.
@ checkers/directio.c
int libcheck_init (struct checker * c)
{
unsigned long pgsize = getpagesize();
struct directio_context * ct;
long flags;
ct = malloc(
if (!ct)
return 1;
memset(ct, 0, sizeof(struct directio_context));
if (io_setup(1, &ct->ioctx) != 0) {
free(ct);
return 1;
}
<...>
The syscall is failing w/ EAGAIN
# grep ^io_setup multipath_
io_setup(1, 0x100163c9130) = -1 EAGAIN (Resource temporarily unavailable)
io_setup(1, 0x10015bae2c0) = -1 EAGAIN (Resource temporarily unavailable)
io_setup(1, 0x100164d65a0) = -1 EAGAIN (Resource temporarily unavailable)
io_setup(1, 0x10016429f20) = -1 EAGAIN (Resource temporarily unavailable)
io_setup(1, 0x100163535c0) = -1 EAGAIN (Resource temporarily unavailable)
io_setup(1, 0x10016368510) = -1 EAGAIN (Resource temporarily unavailable)
<...>
According to the manpage (man 2 io_setup)
NAME
io_setup - create an asynchronous I/O context
DESCRIPTION
The io_setup() system call creates an asynchronous I/O context suitable for concurrently processing nr_events operations. <...>
ERRORS
EAGAIN The specified nr_events exceeds the user's limit of available events, as defined in /proc/sys/
On luckyv1:
root@luckyv1:
65536
root@luckyv1:
130560
According to linux's Documentation/
aio-nr & aio-max-nr:
aio-nr is the running total of the number of events specified on the
io_setup system call for all currently active aio contexts. If aio-nr
reaches aio-max-nr then io_setup will fail with EAGAIN. Note that
raising aio-max-nr does not result in the pre-allocation or re-sizing
of any kernel data structures.
Interestingly, aio-nr is greater than aio-max-nr. Hm.
Increased aio-max-nr to 262144, and could get some more maps created.
== Comment: #8 - Mauricio Faria De Oliveira <email address hidden> - 2016-09-27 18:56:08 ==
This attached test-case demonstrates that for each io_setup() request of 1 nr_event, actually 1280 seem to be allocated.
root@luckyv1:
root@luckyv1:
0
root@luckyv1:
[1] 12352
io_setup rc = 0
sleeping 10 seconds...
root@luckyv1:
1280
<...>
io_destroy rc = 0
[1]+ Done ./io_setup
root@luckyv1:
0
== Comment: #45 - Mauricio Faria De Oliveira <email address hidden> - 2017-09-19 18:32:10 ==
Verification of this commit with the linux-hwe-edge kernel in -proposed,
using the attached test-case "io_setup_v2.c"
commit 2a8a98673c13cb2
Author: Mauricio Faria de Oliveira <email address hidden>
Date: Wed Jul 5 10:53:16 2017 -0300
fs: aio: fix the increment of aio-nr and counting against aio-max-nr
Test-case (attached)
$ sudo apt-get install gcc libaio-dev
$ gcc -o io_setup_v2 io_setup_v2.c -laio
Original kernel:
- Only 409 io_contexts could be allocated,
but that took 130880 [ div by 2, per bug] = 65440 slots out of 65535
$ uname -rv
4.11.
$ ./io_setup_v2 1 65536
nr_events: 1, nr_requests: 65536
rc = -11, i = 409
^Z
[1]+ Stopped ./io_setup_v2 1 65536
$ cat /proc/sys/fs/aio-nr
130880
$ cat /proc/sys/
65536
$ kill %%
Patched kernel:
- Now 65515 io_contexts could be allocated out of 65535 (much better)
(and reporting correctly, without div by 2.)
$ uname -rv
4.11.
$ ./io_setup_v2 1 65536
nr_events: 1, nr_requests: 65536
rc = -12, i = 65515
^Z
[1]+ Stopped ./io_setup_v2 1 65536
$ cat /proc/sys/fs/aio-nr
65515
$ kill %%
CVE References
| bugproxy (bugproxy) wrote output of multipath -ll with verbosity increased to 6 | #1 |
| tags: | added: architecture-ppc64le bugnameltc-146489 severity-critical targetmilestone-inin16043 |
| bugproxy (bugproxy) wrote test-case: io_setup.c | #2 |
| bugproxy (bugproxy) wrote test-case: io_setup_v2.c | #3 |
| Changed in ubuntu: | |
| assignee: | nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
| affects: | ubuntu → kernel-package (Ubuntu) |
| Changed in ubuntu-power-systems: | |
| assignee: | nobody → Canonical Kernel Team (canonical-kernel-team) |
| importance: | Undecided → Critical |
| Mauricio Faria de Oliveira (mfo) wrote | #6 |
| Changed in kernel-package (Ubuntu): | |
| importance: | Undecided → Critical |
| tags: | added: kernel-da-key |
| affects: | kernel-package (Ubuntu) → linux (Ubuntu) |
| Changed in linux (Ubuntu): | |
| status: | New → Triaged |
| Changed in linux (Ubuntu Zesty): | |
| status: | New → Triaged |
| Changed in linux (Ubuntu Xenial): | |
| status: | New → Triaged |
| Changed in linux (Ubuntu Zesty): | |
| importance: | Undecided → Critical |
| Changed in linux (Ubuntu Xenial): | |
| importance: | Undecided → Critical |
| Changed in ubuntu-power-systems: | |
| status: | New → Triaged |
| bugproxy (bugproxy) wrote output of multipath -ll with verbosity increased to 6 | #7 |
| bugproxy (bugproxy) wrote test-case: io_setup.c | #8 |
| bugproxy (bugproxy) wrote test-case: io_setup_v2.c | #9 |
| Changed in linux (Ubuntu Artful): | |
| status: | Triaged → Fix Committed |
| Launchpad Janitor (janitor) wrote | #10 |
| Changed in linux (Ubuntu Artful): | |
| status: | Fix Committed → Fix Released |
| Changed in ubuntu-power-systems: | |
| status: | Triaged → In Progress |
| Changed in linux (Ubuntu Zesty): | |
| assignee: | nobody → Canonical Kernel Team (canonical-kernel-team) |
| tags: | added: triage-a |
| tags: | removed: bugnameltc-146489 kernel-da-key severity-critical triage-a |
| Changed in linux (Ubuntu Zesty): | |
| status: | Triaged → Fix Committed |
| Changed in linux (Ubuntu Xenial): | |
| status: | Triaged → Fix Committed |
| Changed in ubuntu-power-systems: | |
| status: | In Progress → Fix Committed |
| Kleber Sacilotto de Souza (kleber-souza) wrote | #11 |
| tags: | added: verification-needed-xenial |
| tags: | added: verification-needed-zesty |
| Kleber Sacilotto de Souza (kleber-souza) wrote | #12 |
| Kleber Sacilotto de Souza (kleber-souza) wrote | #13 |
| Launchpad Janitor (janitor) wrote | #14 |
| Changed in linux (Ubuntu Zesty): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote | #15 |
| Changed in linux (Ubuntu Xenial): | |
| status: | Fix Committed → Fix Released |
| Changed in ubuntu-power-systems: | |
| status: | Fix Committed → Fix Released |
| Mauricio Faria de Oliveira (mfo) wrote | #16 |
| tags: |
added: verification-done-xenial removed: verification-needed-xenial |
| Mauricio Faria de Oliveira (mfo) wrote | #17 |
| tags: |
added: verification-done-zesty removed: verification-needed-zesty |
| tags: |
added: bugnameltc-146489 severity-critical removed: verification-done-xenial verification-done-zesty |
Default Comment by Bridge