| 2010-09-21 16:20:06 |
John Morrissey |
bug |
|
|
added bug |
| 2010-09-21 16:20:06 |
John Morrissey |
attachment added |
|
udev debug output https://bugs.launchpad.net/bugs/644489/+attachment/1618717/+files/udev-out |
|
| 2010-09-21 16:22:43 |
Matthew Ernisse |
bug |
|
|
added subscriber Matthew Ernisse |
| 2010-09-21 16:50:09 |
John Morrissey |
attachment added |
|
udevadm monitor -e https://bugs.launchpad.net/ubuntu/+source/udev/+bug/644489/+attachment/1618827/+files/udevadm-monitor-events |
|
| 2010-09-21 17:00:55 |
John Morrissey |
affects |
udev (Ubuntu) |
multipath-tools (Ubuntu) |
|
| 2010-09-22 08:32:11 |
Thierry Carrez |
multipath-tools (Ubuntu): importance |
Undecided |
High |
|
| 2010-09-22 08:32:11 |
Thierry Carrez |
multipath-tools (Ubuntu): status |
New |
Confirmed |
|
| 2011-04-06 21:50:43 |
Serge Hallyn |
bug |
|
|
added subscriber Serge Hallyn |
| 2011-04-08 23:20:51 |
John Morrissey |
attachment added |
|
multipathd(8) verbose output https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2003108/+files/multipathd.20110408 |
|
| 2011-04-11 12:55:09 |
Serge Hallyn |
bug task added |
|
udev (Ubuntu) |
|
| 2011-04-12 23:44:09 |
John Morrissey |
attachment added |
|
udevd-debug.gz https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2027325/+files/udevd-debug.gz |
|
| 2011-04-15 03:53:30 |
Serge Hallyn |
multipath-tools (Ubuntu): assignee |
|
Peter Petrakis (peter-petrakis) |
|
| 2011-04-28 10:47:32 |
Olivier Duquesne |
bug |
|
|
added subscriber Olivier Duquesne |
| 2011-05-19 08:45:09 |
krims0n |
bug |
|
|
added subscriber krims0n |
| 2011-06-18 21:00:59 |
Peter Petrakis |
attachment added |
|
updated netapp prio checker https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2174493/+files/mpath_prio_netapp |
|
| 2011-06-19 10:32:06 |
Peter Petrakis |
attachment added |
|
preliminary patch, awaiting upstream feedback https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2174791/+files/adjust-prioritizer-open-flags-to-avoid-sg-io-side-effects.debdiff |
|
| 2011-06-21 18:44:20 |
Peter Petrakis |
multipath-tools (Ubuntu): status |
Confirmed |
In Progress |
|
| 2011-06-21 18:46:13 |
Peter Petrakis |
udev (Ubuntu): status |
New |
Invalid |
|
| 2011-06-21 19:00:44 |
Peter Petrakis |
attachment added |
|
multipath-tools-eliminate-udev-change-events-lp644489.debdiff https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2177364/+files/multipath-tools-eliminate-udev-change-events-lp644489.debdiff |
|
| 2011-06-21 20:09:25 |
Peter Petrakis |
multipath-tools (Ubuntu): status |
In Progress |
Fix Released |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
nominated for series |
|
Ubuntu Lucid |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
bug task added |
|
multipath-tools (Ubuntu Lucid) |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
bug task added |
|
udev (Ubuntu Lucid) |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
nominated for series |
|
Ubuntu Maverick |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
bug task added |
|
multipath-tools (Ubuntu Maverick) |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
bug task added |
|
udev (Ubuntu Maverick) |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
nominated for series |
|
Ubuntu Natty |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
bug task added |
|
multipath-tools (Ubuntu Natty) |
|
| 2011-06-21 20:15:21 |
Serge Hallyn |
bug task added |
|
udev (Ubuntu Natty) |
|
| 2011-06-21 20:15:40 |
Serge Hallyn |
multipath-tools (Ubuntu Lucid): status |
New |
Confirmed |
|
| 2011-06-21 20:15:44 |
Serge Hallyn |
multipath-tools (Ubuntu Maverick): status |
New |
Confirmed |
|
| 2011-06-21 20:15:50 |
Serge Hallyn |
multipath-tools (Ubuntu Natty): status |
New |
Confirmed |
|
| 2011-06-21 20:16:06 |
Serge Hallyn |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2011-06-21 20:33:30 |
Peter Petrakis |
udev (Ubuntu Lucid): status |
New |
Invalid |
|
| 2011-06-21 20:33:33 |
Peter Petrakis |
udev (Ubuntu Maverick): status |
New |
Invalid |
|
| 2011-06-21 20:33:36 |
Peter Petrakis |
udev (Ubuntu Natty): status |
New |
Invalid |
|
| 2011-06-21 20:34:55 |
Peter Petrakis |
affects |
udev (Ubuntu) |
ubuntu |
|
| 2011-06-21 20:37:14 |
Peter Petrakis |
description |
Binary package hint: udev
udevd constantly changes LUN device node symlinks (devices/LUNs, not the partition nodes) in /dev/disk/by-id. udevd uses ~15% of CPU and system time is using ~50-60%.
For example:
[jwm@syslog01.roch.ny:pts/0 /dev/disk/by-id> ls -l wwn-0x60a98000486e5339576f596675735354 wwn-0x60a98000486e5339576f596675744c36 scsi-360a98000486e5339576f596675735354 scsi-360a98000486e5339576f596675744c36; sleep 1; echo '======'; ls -l wwn-0x60a98000486e5339576f596675735354 wwn-0x60a98000486e5339576f596675744c36 scsi-360a98000486e5339576f596675735354 scsi-360a98000486e5339576f596675744c36
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675735354 -> ../../sde
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675744c36 -> ../../sdf
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675735354 -> ../../sde
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675744c36 -> ../../sdf
======
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675735354 -> ../../sdg
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675744c36 -> ../../sdh
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675735354 -> ../../sdg
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675744c36 -> ../../sdh
All other device nodes stay the same, such as the device nodes for the partitions:
[jwm@syslog01.roch.ny:pts/0 /dev/disk/by-id> ls -l scsi-360a98000486e5339576f596675735354-part1; sleep 1; echo '======'; ls -l scsi-360a98000486e5339576f596675735354-part1
lrwxrwxrwx 1 root root 10 2010-09-21 15:47 scsi-360a98000486e5339576f596675735354-part1 -> ../../sdg1
======
lrwxrwxrwx 1 root root 10 2010-09-21 15:47 scsi-360a98000486e5339576f596675735354-part1 -> ../../sdg1
I'm not entirely sure whether this is udev's problem or something related to multipathing. Our most recent experience with multipathing is the last LTS release (hardy), which doesn't exhibit this behavior given similar configurations.
[jwm@syslog01.roch.ny:pts/0 ~> sudo multipath -ll
rootvol (360a98000486e5339576f596675735354) dm-1 NETAPP ,LUN
[size=36G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=8][active]
\_ 2:0:2:0 sda 8:0 [active][ready]
\_ 3:0:2:0 sde 8:64 [active][ready]
\_ round-robin 0 [prio=2][enabled]
\_ 3:0:3:0 sdg 8:96 [active][ready]
\_ 2:0:3:0 sdc 8:32 [active][ready]
syslog-data (360a98000486e5339576f596675744c36) dm-0 NETAPP ,LUN
[size=1.0T][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=8][active]
\_ 2:0:2:1 sdb 8:16 [active][ready]
\_ 3:0:2:1 sdf 8:80 [active][ready]
\_ round-robin 0 [prio=2][enabled]
\_ 3:0:3:1 sdh 8:112 [active][ready]
\_ 2:0:3:1 sdd 8:48 [active][ready]
[jwm@syslog01.roch.ny:pts/0 ~> cat /etc/multipath.conf
multipaths {
multipath {
wwid 360a98000486e5339576f596675735354
alias rootvol
}
multipath {
wwid 360a98000486e5339576f596675744c36
alias syslog-data
}
}
devices {
device {
vendor "NETAPP "
product "LUN "
path_checker tur
path_grouping_policy group_by_prio
prio_callout "/sbin/mpath_prio_netapp /dev/%n"
failback immediate
rr_min_io 128
no_path_retry queue
}
} |
= SRU Justification =
== Impact ==
Multipath-tools is inadvertedly generating UDEV CHANGE events for the SD
block devices under it's control. These change events feedback into the udev
rules, increasing cpu utlilzation, and ruining the multipath aliasing feature,
which allows one to rename a multipath path from a series of letters and
numbers, to a human readable label. It gives users the impression that
the SAN is unstable.
== Solution ==
Change the open() flags in the priority checkers to read only from read write, this
stops sd from generating a change event after the file descriptor has been
closed.
Patch: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2177364/+files/multipath-tools-eliminate-udev-change-events-lp644489.debdiff
== Reproduction ==
Is easy, and doesn't even require a SAN. Since we're dealing with simple SCSI
inquiry cmds any block device will do. Simply install multipath-tools and
execute one of the priority checkers like so:
/sbin/mpath_prio_emc /dev/sda
also, have a window open monitoring udev, udevadm monitor, ensure
no change events to that block device are occurring before hand.
TEST CASE:
root@kickseed:~# udevadm monitor &
[1] 16950
root@kickseed:~# monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
root@kickseed:~#
root@kickseed:~# /sbin/mpath_prio_emc /dev/sda
query command indicates error0
root@kickseed:~# KERNEL[1308688009.806317] change /devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:0/expander-0:0/port-0:0:1/end_device-0:0:1/target0:0:0/0:0:0:0/block/sda (block)
UDEV [1308688009.823569] change /devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:0/expander-0:0/port-0:0:1/end_device-0:0:1/target0:0:0/0:0:0:0/block/sda (block)
root@kickseed:~#
root@kickseed:~#
root@kickseed:~# /sbin/mpath_prio_alua /dev/sda
130
mpath_prio_alua doesn't generate any change events since it's open
flags do not include O_RDRW to begin with.
== regression potential ==
None, it's broken to begin with.
--------------------------
Binary package hint: udev
udevd constantly changes LUN device node symlinks (devices/LUNs, not the partition nodes) in /dev/disk/by-id. udevd uses ~15% of CPU and system time is using ~50-60%.
For example:
[jwm@syslog01.roch.ny:pts/0 /dev/disk/by-id> ls -l wwn-0x60a98000486e5339576f596675735354 wwn-0x60a98000486e5339576f596675744c36 scsi-360a98000486e5339576f596675735354 scsi-360a98000486e5339576f596675744c36; sleep 1; echo '======'; ls -l wwn-0x60a98000486e5339576f596675735354 wwn-0x60a98000486e5339576f596675744c36 scsi-360a98000486e5339576f596675735354 scsi-360a98000486e5339576f596675744c36
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675735354 -> ../../sde
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675744c36 -> ../../sdf
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675735354 -> ../../sde
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675744c36 -> ../../sdf
======
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675735354 -> ../../sdg
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675744c36 -> ../../sdh
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675735354 -> ../../sdg
lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675744c36 -> ../../sdh
All other device nodes stay the same, such as the device nodes for the partitions:
[jwm@syslog01.roch.ny:pts/0 /dev/disk/by-id> ls -l scsi-360a98000486e5339576f596675735354-part1; sleep 1; echo '======'; ls -l scsi-360a98000486e5339576f596675735354-part1
lrwxrwxrwx 1 root root 10 2010-09-21 15:47 scsi-360a98000486e5339576f596675735354-part1 -> ../../sdg1
======
lrwxrwxrwx 1 root root 10 2010-09-21 15:47 scsi-360a98000486e5339576f596675735354-part1 -> ../../sdg1
I'm not entirely sure whether this is udev's problem or something related to multipathing. Our most recent experience with multipathing is the last LTS release (hardy), which doesn't exhibit this behavior given similar configurations.
[jwm@syslog01.roch.ny:pts/0 ~> sudo multipath -ll
rootvol (360a98000486e5339576f596675735354) dm-1 NETAPP ,LUN
[size=36G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=8][active]
\_ 2:0:2:0 sda 8:0 [active][ready]
\_ 3:0:2:0 sde 8:64 [active][ready]
\_ round-robin 0 [prio=2][enabled]
\_ 3:0:3:0 sdg 8:96 [active][ready]
\_ 2:0:3:0 sdc 8:32 [active][ready]
syslog-data (360a98000486e5339576f596675744c36) dm-0 NETAPP ,LUN
[size=1.0T][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=8][active]
\_ 2:0:2:1 sdb 8:16 [active][ready]
\_ 3:0:2:1 sdf 8:80 [active][ready]
\_ round-robin 0 [prio=2][enabled]
\_ 3:0:3:1 sdh 8:112 [active][ready]
\_ 2:0:3:1 sdd 8:48 [active][ready]
[jwm@syslog01.roch.ny:pts/0 ~> cat /etc/multipath.conf
multipaths {
multipath {
wwid 360a98000486e5339576f596675735354
alias rootvol
}
multipath {
wwid 360a98000486e5339576f596675744c36
alias syslog-data
}
}
devices {
device {
vendor "NETAPP "
product "LUN "
path_checker tur
path_grouping_policy group_by_prio
prio_callout "/sbin/mpath_prio_netapp /dev/%n"
failback immediate
rr_min_io 128
no_path_retry queue
}
} |
|
| 2011-06-21 21:40:15 |
Peter Petrakis |
ubuntu: status |
Invalid |
Confirmed |
|
| 2011-06-21 21:40:20 |
Peter Petrakis |
Ubuntu Lucid: status |
Invalid |
Confirmed |
|
| 2011-06-21 21:40:23 |
Peter Petrakis |
Ubuntu Maverick: status |
Invalid |
Confirmed |
|
| 2011-06-21 21:40:25 |
Peter Petrakis |
Ubuntu Natty: status |
Invalid |
Confirmed |
|
| 2011-06-21 21:52:22 |
krims0n |
removed subscriber krims0n |
|
|
|
| 2011-06-24 16:04:14 |
Martin Pitt |
ubuntu: status |
Confirmed |
Invalid |
|
| 2011-06-24 16:04:17 |
Martin Pitt |
Ubuntu Lucid: status |
Confirmed |
Invalid |
|
| 2011-06-24 16:04:19 |
Martin Pitt |
Ubuntu Maverick: status |
Confirmed |
Invalid |
|
| 2011-06-24 16:04:22 |
Martin Pitt |
Ubuntu Natty: status |
Confirmed |
Invalid |
|
| 2011-06-24 16:04:36 |
Martin Pitt |
multipath-tools (Ubuntu Natty): status |
Confirmed |
Fix Committed |
|
| 2011-06-24 16:04:39 |
Martin Pitt |
bug |
|
|
added subscriber SRU Verification |
| 2011-06-24 16:04:44 |
Martin Pitt |
tags |
|
verification-needed |
|
| 2011-06-24 17:25:44 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/natty-proposed/multipath-tools |
|
| 2011-06-25 08:03:23 |
Martin Pitt |
tags |
verification-needed |
verification-done verification-done-natty |
|
| 2011-06-26 14:56:27 |
Jorge Salamero Sanz |
bug |
|
|
added subscriber Jorge Salamero Sanz |
| 2011-06-28 06:48:02 |
Clint Byrum |
multipath-tools (Ubuntu Maverick): status |
Confirmed |
Fix Committed |
|
| 2011-06-28 06:48:07 |
Clint Byrum |
tags |
verification-done verification-done-natty |
verification-done-natty |
|
| 2011-06-28 06:48:09 |
Clint Byrum |
tags |
verification-done-natty |
verification-done-natty verification-needed |
|
| 2011-06-28 06:53:50 |
Clint Byrum |
multipath-tools (Ubuntu Lucid): status |
Confirmed |
Fix Committed |
|
| 2011-06-28 07:16:01 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/maverick-proposed/multipath-tools |
|
| 2011-06-29 06:32:50 |
Martin Pitt |
tags |
verification-done-natty verification-needed |
verification-done |
|
| 2011-07-05 13:40:45 |
Launchpad Janitor |
multipath-tools (Ubuntu Lucid): status |
Fix Committed |
Fix Released |
|
| 2011-07-05 13:40:53 |
Launchpad Janitor |
multipath-tools (Ubuntu Maverick): status |
Fix Committed |
Fix Released |
|
| 2011-07-05 13:40:57 |
Launchpad Janitor |
multipath-tools (Ubuntu Natty): status |
Fix Committed |
Fix Released |
|
| 2011-07-05 14:22:31 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/lucid-updates/multipath-tools |
|
| 2011-09-19 21:30:13 |
Ubuntu Foundations Team Bug Bot |
tags |
verification-done |
testcase verification-done |
|
