iscsitarget causing kernel lockup
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
iscsitarget (Ubuntu) |
Confirmed
|
High
|
Unassigned |
Bug Description
Under heavy load, ISCSITARGET on Ubuntu 12.04 LTS is crashing repeatedly. The syslog contains;
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568639] ------------[ cut here ]------------
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568669] kernel BUG at /var/lib/
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568708] invalid opcode: 0000 [#1] SMP
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568735] CPU 0
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568747] Modules linked in: iscsi_trgt(O) xen_fbfront joydev mac_hid fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront lp parport [last unloaded: iscsi_trgt]
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568861]
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568873] Pid: 1464, comm: istd1 Tainted: G O 3.2.0-32-generic #51-Ubuntu
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568917] RIP: e030:[<
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568968] RSP: e02b:ffff88003b
Nov 13 11:05:38 iscsi0 kernel: [ 3330.568995] RAX: 0000000000000000 RBX: ffff8800207c5618 RCX: 0000000000000001
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569030] RDX: 0000000000000000 RSI: ffff88003b0a1f58 RDI: ffff88003b0a1f58
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569065] RBP: ffff88003b0a1d90 R08: 000000000000000a R09: 0000000000000000
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569100] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880003e72000
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569134] R13: ffff880003e70000 R14: 0000000000000000 R15: ffff880004188578
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569177] FS: 00007f08709347c
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569217] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569246] CR2: 00007f7d186b0000 CR3: 000000003cf25000 CR4: 0000000000002660
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569315] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569350] Process istd1 (pid: 1464, threadinfo ffff88003b0a0000, task ffff88003a7b4500)
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569389] Stack:
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569401] 0000000000000138 0000000000000138 ffff88003b0a1d30 ffff880000000000
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569447] ffff88003b0a1910 0000000000000001 ffff88003b0a1d50 ffffffff8165ae3e
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569494] 0000047c05a95a00 00000000a0064673 ffff88003b0a1d80 ffff880003e72000
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569540] Call Trace:
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569557] [<ffffffff8165a
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569588] [<ffffffffa0067
Nov 13 11:05:38 iscsi0 kernel: [ 3330.569620] [<ffffffff815aa
Nov 13 11:05:38 iscsi0 kernel: [ 3330.570514] [<ffffffff8104c
Nov 13 11:05:38 iscsi0 kernel: [ 3330.571409] [<ffffffff8165b
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572314] [<ffffffffa0067
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] [<ffffffff8108a
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] [<ffffffff81665
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] [<ffffffff81663
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] [<ffffffff8165b
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] [<ffffffff81665
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] Code: 89 b9 fa ff ff 48 c7 c1 84 40 07 a0 ba 1d 04 00 00 48 c7 c6 d8 27 07 a0 48 c7 c7 10 28 07 a0 31 c0 e8 f1 b9 5d e1 e8 37 8d 5d e1 <0f> 0b 80 7b 39 00 0f 89 01 02 00 00 44 8b 5b 74 45 85 db 0f 85
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] RIP [<ffffffffa0066
Nov 13 11:05:38 iscsi0 kernel: [ 3330.572320] RSP <ffff88003b0a1d10>
Nov 13 11:05:38 iscsi0 kernel: [ 3330.583226] ---[ end trace 37722237d92cd12b ]---
I have this error on both a physical box (2x 4-core CPU, 128GB RAM) and a virtual machine (1 vcpu, 1GB RAM). Have tried both blockio and fileio modes but the problem persists. Have tried changing RAM and CPU sizing to no avail.
The problem is seemingly random but usually occurs under a decent load.
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: iscsitarget 1.4.20.2-5ubuntu3.1
ProcVersionSign
Uname: Linux 3.2.0-32-generic x86_64
ApportVersion: 2.0.1-0ubuntu14
Architecture: amd64
Date: Tue Nov 13 11:45:52 2012
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: iscsitarget
UpgradeStatus: No upgrade log present (probably fresh install)
modified.
mtime.conffile.
mtime.conffile.
mtime.conffile.
mtime.conffile.
I don't know for sure if this is pertinent;
The problem didn't occur until the LUN started to fill up with data. There are four LUNs in total spread across two client machines. I have 38TB of local storage which I have carved up into a 10TB LUN and a 9TB LUN, with two LUNs presented to each host.
Is this perhaps a size limit? The problem was not extant until I had more than 5TB on a single LUN. Is there an addressing limit which is maybe causing this crash once we exceed that limit?