| 2012-09-20 09:45:13 |
dann frazier |
bug |
|
|
added bug |
| 2012-09-20 09:45:56 |
dann frazier |
attachment added |
|
iscsiadm.strace https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1053306/+attachment/3324748/+files/iscsiadm.strace |
|
| 2012-09-20 09:47:20 |
dann frazier |
attachment added |
|
strace attached to iscsid during period where iscsiadm command runs https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1053306/+attachment/3324749/+files/iscsid.strace |
|
| 2012-09-20 11:12:35 |
dann frazier |
bug watch added |
|
https://bugzilla.redhat.com/show_bug.cgi?id=624344 |
|
| 2012-09-20 21:50:16 |
Narinder Gupta |
bug |
|
|
added subscriber Narinder Gupta |
| 2012-09-21 16:18:43 |
Brian Murray |
tags |
|
precise |
|
| 2012-09-21 16:19:35 |
Brian Murray |
open-iscsi (Ubuntu): importance |
Undecided |
High |
|
| 2012-09-21 16:20:07 |
Brian Murray |
bug task added |
|
open-iscsi (Fedora) |
|
| 2012-09-21 20:35:38 |
Leonardo Borda |
bug |
|
|
added subscriber Leonardo Borda |
| 2012-09-24 01:49:52 |
dann frazier |
description |
We occasionally see iscsiadm hang indefinitely on a system. It is difficult to reproduce, there maybe months between occurrences. This happened again today, and we collected the following information.
We captured an strace of a failing iscsiadm discover process, I'll attach it here. All iscsiadm processes that are hung are also in recvmsg() when we attach strace.
It was observed that there was one iscsid process - where normally there is 2. The pids are normally consecutive, so I presume one forks the other before daemonizing.
syslog contains the following messages from iscsid:
Sep 20 15:55:57 cn5 iscsid: semop up failed 4
Sep 20 15:56:05 cn5 iscsid: semop down failed
Sep 20 15:56:23 cn5 iscsid: semop up failed 4
Sep 20 16:02:23 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:32 cn5 iscsid: semop up failed 4
Sep 20 18:01:49 cn5 iscsid: semop up failed 4
Sep 20 18:03:56 cn5 iscsid: semop up failed 4
Sep 20 18:05:03 cn5 iscsid: semop up failed 4
This sounds similar to the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=676804, but note the different errno. |
[Impact]
iscsid can hang or crash
[Test Case]
1) This issue is exacerbated by increased logging output. To configure open-iscsi to emit the maximum amount of messages, edit /etc/init.d/open-iscsi and change the line:
start-stop-daemon --start --quiet --exec $DAEMON
to:
start-stop-daemon --start --quiet --exec $DAEMON -- -d 8
On your storage server, create multiple LUN targets and expose them to your Ubuntu system. I recommend 3 or more.
Edit the attached repro.sh script to set the TARGET variable to point to the correct IP of your storage server. Execute this script on your Ubuntu system. When an error occurs, this script should hang or begin emitting errors.
[Regression Potential]
I would expect regressions here to have the same symptoms of the original bug - e.g. hangs, crashes.
Original report follows.
We occasionally see iscsiadm hang indefinitely on a system. It is difficult to reproduce, there maybe months between occurrences. This happened again today, and we collected the following information.
We captured an strace of a failing iscsiadm discover process, I'll attach it here. All iscsiadm processes that are hung are also in recvmsg() when we attach strace.
It was observed that there was one iscsid process - where normally there is 2. The pids are normally consecutive, so I presume one forks the other before daemonizing.
syslog contains the following messages from iscsid:
Sep 20 15:55:57 cn5 iscsid: semop up failed 4
Sep 20 15:56:05 cn5 iscsid: semop down failed
Sep 20 15:56:23 cn5 iscsid: semop up failed 4
Sep 20 16:02:23 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:26 cn5 iscsid: semop up failed 4
Sep 20 16:02:32 cn5 iscsid: semop up failed 4
Sep 20 18:01:49 cn5 iscsid: semop up failed 4
Sep 20 18:03:56 cn5 iscsid: semop up failed 4
Sep 20 18:05:03 cn5 iscsid: semop up failed 4
This sounds similar to the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=676804, but note the different errno. |
|
| 2012-09-24 01:52:15 |
dann frazier |
attachment added |
|
sample reproduction script https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1053306/+attachment/3336393/+files/repro.sh |
|
| 2012-09-24 02:09:06 |
Launchpad Janitor |
branch linked |
|
lp:~dannf/open-iscsi/lp1053306 |
|
| 2012-09-25 06:01:18 |
Ante Karamatić |
nominated for series |
|
Ubuntu Precise |
|
| 2012-09-25 06:01:54 |
dann frazier |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2012-09-25 06:03:37 |
Ante Karamatić |
bug |
|
|
added subscriber Ante Karamatić |
| 2012-10-01 17:55:07 |
Adam Stokes |
branch linked |
|
lp:~adam-stokes/ubuntu/precise/open-iscsi/fix-lp-1053306 |
|
| 2012-10-01 17:56:13 |
Adam Stokes |
branch unlinked |
lp:~adam-stokes/ubuntu/precise/open-iscsi/fix-lp-1053306 |
|
|
| 2012-10-01 17:58:09 |
Adam Stokes |
open-iscsi (Ubuntu): milestone |
|
ubuntu-12.04.2 |
|
| 2012-10-01 17:59:07 |
Adam Stokes |
bug |
|
|
added subscriber Adam Stokes |
| 2012-10-01 19:05:52 |
dann frazier |
attachment added |
|
proposed debdiff https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1053306/+attachment/3362529/+files/open-iscsi_2.0.871-0ubuntu9_to_2.0.871-0ubuntu9.1.debdiff |
|
| 2012-10-01 19:55:14 |
dann frazier |
branch linked |
|
lp:~dannf/ubuntu/precise/open-iscsi/lp1053306 |
|
| 2012-10-02 13:23:35 |
Adam Stokes |
open-iscsi (Ubuntu): assignee |
|
Adam Stokes (adam-stokes) |
|
| 2012-10-02 13:23:38 |
Adam Stokes |
open-iscsi (Ubuntu): status |
New |
In Progress |
|
| 2012-10-03 14:34:32 |
Marc Deslauriers |
bug task added |
|
open-iscsi (Ubuntu Precise) |
|
| 2012-10-03 14:34:41 |
Marc Deslauriers |
nominated for series |
|
Ubuntu Quantal |
|
| 2012-10-03 14:34:41 |
Marc Deslauriers |
bug task added |
|
open-iscsi (Ubuntu Quantal) |
|
| 2012-10-03 14:34:56 |
Marc Deslauriers |
open-iscsi (Ubuntu Quantal): status |
In Progress |
Fix Released |
|
| 2012-10-03 14:35:00 |
Marc Deslauriers |
open-iscsi (Ubuntu Precise): status |
New |
Confirmed |
|
| 2012-10-03 14:35:02 |
Marc Deslauriers |
open-iscsi (Ubuntu Precise): importance |
Undecided |
High |
|
| 2012-10-03 14:41:23 |
Adam Stokes |
open-iscsi (Ubuntu Quantal): milestone |
ubuntu-12.04.2 |
|
|
| 2012-10-03 14:41:25 |
Adam Stokes |
open-iscsi (Ubuntu Precise): milestone |
|
ubuntu-12.04.2 |
|
| 2012-10-03 14:41:27 |
Adam Stokes |
open-iscsi (Ubuntu Precise): assignee |
|
Adam Stokes (adam-stokes) |
|
| 2012-10-03 15:29:23 |
Adam Stokes |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2012-10-05 13:30:23 |
James Page |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2012-10-15 20:19:41 |
Chris J Arges |
bug |
|
|
added subscriber Sustaining Engineering |
| 2012-10-16 11:04:04 |
Daniel Holbach |
open-iscsi (Ubuntu Precise): status |
Confirmed |
In Progress |
|
| 2012-10-16 14:50:51 |
Colin Watson |
open-iscsi (Ubuntu Precise): status |
In Progress |
Fix Committed |
|
| 2012-10-16 14:50:54 |
Colin Watson |
bug |
|
|
added subscriber SRU Verification |
| 2012-10-16 14:50:56 |
Colin Watson |
tags |
precise |
precise verification-needed |
|
| 2012-10-16 15:14:41 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-branches/ubuntu/precise/open-iscsi/precise-proposed |
|
| 2012-10-16 19:57:41 |
Narinder Gupta |
tags |
precise verification-needed |
precise verification-done |
|
| 2012-10-25 20:17:42 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2012-10-25 20:18:10 |
Launchpad Janitor |
open-iscsi (Ubuntu Precise): status |
Fix Committed |
Fix Released |
|
| 2014-02-07 15:46:37 |
Curtis Hovey |
removed subscriber Registry Administrators |
|
|
|
| 2017-10-28 03:26:24 |
Bug Watch Updater |
open-iscsi (Fedora): status |
Unknown |
Fix Released |
|
| 2017-10-28 03:26:24 |
Bug Watch Updater |
open-iscsi (Fedora): importance |
Unknown |
High |
|
