Ubuntu 18.04 - IO Hang on some namespaces when running HTX with 16 namespaces (Bolt / NVMe)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
The Ubuntu-power-systems project |
Fix Released
|
High
|
Canonical Kernel Team | ||
linux (Ubuntu) |
Fix Released
|
High
|
Joseph Salisbury | ||
Bionic |
Fix Released
|
High
|
Joseph Salisbury |
Bug Description
---Problem Description---
We are seeing similar IO Hang on some namespaces when running HTX 16 namespaces on Ubuntu18.04
---uname output---
Linux ltciofvtr-spoon4 4.15.0-10-generic #11-Ubuntu SMP Tue Feb 13 18:21:52 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
---Additional Hardware Info---
(Bolt / NVMe)0003:01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller 172Xa [144d:a822] (rev 01)
Machine Type = AC922
---Steps to Reproduce---
1> Install Ubuntu18.04 , upgrade to 4.15.0-10 kernel
2> Install htxubuntu-472.deb
3> make sure you create name spaces
#!/bin/bash
device=/dev/nvme0
echo $device
nvme format $device
nvme set-feature $device -f 0x0b --value=0x0100
nvme delete-ns $device -n 0xFFFFFFFF
sleep 5
nvme list
nvme get-log $device -l 200 -i 4
max=`nvme id-ctrl $device | grep ^nn | awk '{print $NF}'`
for i in $(eval echo {1..$max})
do
echo $i
nvme create-ns $device --nsze=7000000 --ncap=7000000 --flbas=0 --dps=0
nvme attach-ns $device --namespace-id=$i --controllers=`nvme list-ctrl $device | awk -F: '{print $2}'`
sleep 2
nvme get-log $device -l 200 -i 4
sleep 2
done
nvme list
3> run mdt.hd on those namespaces
Contact Information = <email address hidden>
Stack trace output:
------
-------
Device id:/dev/nvme0n8
Timestamp:Feb 20 16:57:30 2018
err=ffffffff
sev=1
Exerciser Name:hxestorage
Serial No:Not Available
Part No:Not Available
Location:Not Available
FRU Number:Not Available
Device:Not Available
Error Text:Hardware Exerciser stopped on error
-------
-------
Device id:/dev/nvme0n10
Timestamp:Feb 20 16:57:36 2018
err=ffffffff
sev=1
Exerciser Name:hxestorage
Serial No:Not Available
Part No:Not Available
Location:Not Available
FRU Number:Not Available
Device:Not Available
Error Text:Hung I/O alert! Segment table-1, Detected 1 I/O(s) hung.
Current time: 1519163856; hang criteria: 600 secs, Hard hang threshold: 3
Process ID: 0x8161
1st lba Blocks Kernel Hang Duration
(Hex) (Hex) Thread Cnt (Secs)
** Threshold of 1800 secs on one or more I/Os exceeded!
0x5ae08b 8 7e0457eaf180 4 4800
-------
-------
Device id:/dev/nvme0n10
Timestamp:Feb 20 16:57:36 2018
err=ffffffff
sev=1
Exerciser Name:hxestorage
Serial No:Not Available
Part No:Not Available
Location:Not Available
FRU Number:Not Available
Device:Not Available
Error Text:Hardware Exerciser stopped on error
-------
-------
Device id:/dev/nvme0n4
Timestamp:Feb 20 17:14:19 2018
err=ffffffff
sev=4
Exerciser Name:hxestorage
Serial No:Not Available
Part No:Not Available
Location:Not Available
FRU Number:Not Available
Device:Not Available
Error Text:Hung I/O alert! Segment table-1, Detected 1 I/O(s) hung.
Current time: 1519164859; hang criteria: 600 secs, Hard hang threshold: 3
Process ID: 0x815b
1st lba Blocks Kernel Hang Duration
(Hex) (Hex) Thread Cnt (Secs)
0x398a7e 2 71d5affff180 3 3000
-------
[17643.202114] INFO: task hxestorage:39744 blocked for more than 120 seconds.
[17643.202180] Not tainted 4.15.0-10-generic #11-Ubuntu
[17643.202224] "echo 0 > /proc/sys/
[17643.202342] hxestorage D 0 39744 3424 0x00040000
[17643.202346] Call Trace:
[17643.202352] [c00020382bc4b660] [c00020382bc4b6b0] 0xc00020382bc4b6b0 (unreliable)
[17643.202360] [c00020382bc4b830] [c00000000001c080] __switch_
[17643.202364] [c00020382bc4b890] [c000000000cfce84] __schedule+
[17643.202366] [c00020382bc4b960] [c000000000cfd710] schedule+0x40/0xc0
[17643.202370] [c00020382bc4b980] [c00000000014dffc] io_schedule+
[17643.202376] [c00020382bc4b9b0] [c00000000042bf94] __blkdev_
[17643.202379] [c00020382bc4bae0] [c00000000042c500] blkdev_
[17643.202384] [c00020382bc4bbb0] [c0000000002dc1f8] generic_
[17643.202387] [c00020382bc4bc20] [c0000000002dc47c] __generic_
[17643.202391] [c00020382bc4bc80] [c00000000042da3c] blkdev_
[17643.202394] [c00020382bc4bcf0] [c0000000003cc3f4] new_sync_
[17643.202397] [c00020382bc4bd80] [c0000000003cfb38] vfs_write+
[17643.202401] [c00020382bc4bdd0] [c0000000003d00b4] SyS_pwrite64+
[17643.202405] [c00020382bc4be30] [c00000000000b184] system_
[17643.202408] INFO: task hxestorage:39748 blocked for more than 120 seconds.
[17643.202519] Not tainted 4.15.0-10-generic #11-Ubuntu
[17643.202587] "echo 0 > /proc/sys/
[17643.202692] hxestorage D 0 39748 3424 0x00040000
[17643.202695] Call Trace:
[17643.202697] [c00020382bc6f660] [c00020382bc6f6b0] 0xc00020382bc6f6b0 (unreliable)
[17643.202701] [c00020382bc6f830] [c00000000001c080] __switch_
[17643.202703] [c00020382bc6f890] [c000000000cfce84] __schedule+
[17643.202705] [c00020382bc6f960] [c000000000cfd710] schedule+0x40/0xc0
[17643.202708] [c00020382bc6f980] [c00000000014dffc] io_schedule+
[17643.202711] [c00020382bc6f9b0] [c00000000042bf94] __blkdev_
[17643.202714] [c00020382bc6fae0] [c00000000042c500] blkdev_
[17643.202717] [c00020382bc6fbb0] [c0000000002dc1f8] generic_
[17643.202720] [c00020382bc6fc20] [c0000000002dc47c] __generic_
[17643.202723] [c00020382bc6fc80] [c00000000042da3c] blkdev_
[17643.202726] [c00020382bc6fcf0] [c0000000003cc3f4] new_sync_
[17643.202729] [c00020382bc6fd80] [c0000000003cfb38] vfs_write+
[17643.202732] [c00020382bc6fdd0] [c0000000003d00b4] SyS_pwrite64+
[17643.202735] [c00020382bc6fe30] [c00000000000b184] system_
[17643.202740] INFO: task hxestorage:39917 blocked for more than 120 seconds.
[17643.202809] Not tainted 4.15.0-10-generic #11-Ubuntu
[17643.202882] "echo 0 > /proc/sys/
[17643.203013] hxestorage D 0 39917 3424 0x00040000
[17643.203015] Call Trace:
[17643.203017] [c00020382bcd3720] [0000003c00000000] 0x3c00000000 (unreliable)
[17643.203021] [c00020382bcd38f0] [c00000000001c080] __switch_
[17643.203023] [c00020382bcd3950] [c000000000cfce84] __schedule+
[17643.203025] [c00020382bcd3a20] [c000000000cfd710] schedule+0x40/0xc0
[17643.203027] [c00020382bcd3a40] [c00000000014dffc] io_schedule+
[17643.203030] [c00020382bcd3a70] [c00000000042bf94] __blkdev_
[17643.203033] [c00020382bcd3ba0] [c00000000042c500] blkdev_
[17643.203036] [c00020382bcd3c70] [c0000000002dbfdc] generic_
[17643.203040] [c00020382bcd3cd0] [c00000000042d1e0] blkdev_
[17643.203043] [c00020382bcd3cf0] [c0000000003cc290] new_sync_
[17643.203046] [c00020382bcd3d80] [c0000000003cf74c] vfs_read+0xbc/0x1b0
[17643.203049] [c00020382bcd3dd0] [c0000000003cffc4] SyS_pread64+
[17643.203052] [c00020382bcd3e30] [c00000000000b184] system_
[17643.203056] INFO: task hxestorage:40049 blocked for more than 120 seconds.
Possible patch being reviewed for this issue:
tags: | added: kernel-da-key |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in ubuntu-power-systems: | |
importance: | Undecided → High |
assignee: | nobody → Canonical Kernel Team (canonical-kernel-team) |
Changed in ubuntu-power-systems: | |
status: | New → Triaged |
tags: | added: triage-g |
Changed in linux (Ubuntu Bionic): | |
status: | Triaged → In Progress |
assignee: | Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Joseph Salisbury (jsalisbury) |
Changed in ubuntu-power-systems: | |
status: | Triaged → In Progress |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Fix Committed |
Changed in ubuntu-power-systems: | |
status: | In Progress → Fix Committed |
Changed in ubuntu-power-systems: | |
status: | Fix Committed → Fix Released |
Default Comment by Bridge