STCOP810:Firestone: frsfp6 EEH on Bluefin does not recover with Ubuntu
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Taco Screen team | ||
Vivid |
Fix Released
|
Undecided
|
Tim Gardner | ||
Wily |
Fix Released
|
Undecided
|
Taco Screen team |
Bug Description
Problem:
==========
Test Case Execution Record:
95613: EEH_Firestone_
Error Injection Method: err_injct_inboundA
Step 1. Start HTX (I used mdt.hdbuster & only ran htx on bluefin disks)
Step 2. Inject EEH error
bluefin is in slot P1-C4 (PCI0004)
echo 0x8000000000000000 > /sys/kernel/
Expected Result: Adapter/SAN disks to recover and htx still run
Actual Result: Adapter did not recover... continuous EEH errors until limit of 6 is reached in 1 hour
There're two patches: one for skiboot firmware and another patch, which has been in upstream, was missed in ubuntu distro (at least 15.04). The skiboot patch has been merged to upstream.
c7192a4 PHB3: Fix wrong PE number in error injection (skiboot)
2aa5cf9 powerpc/eeh: Fix missed PE#0 on P7IOC (linux)
If I'm correct, I think this bug needs to be mirrored so that the Linux patch (commit 2aa5cf9) can be backported to ubuntu distro. With the patch backported to ubuntu 15.04, EEH works fine on Broadcom adapter (not exactly the one where the bug was reported initially):
root@fstn2-p1:/# dmesg | grep EEH
[ 0.216919] EEH: PowerNV platform initialized
[ 0.570606] EEH: devices created
[ 1.302482] EEH: PCI Enhanced I/O Error Handling Enabled
[ 90.566761] EEH: PHB location: Slot1
[ 90.567503] EEH: Frozen PHB#4-PE#0 detected
[ 90.567673] EEH: PE location: Slot1, PHB location: Slot1
[ 90.567930] EEH: Detected PCI bus error on PHB#4-PE#0
[ 90.567935] EEH: This PCI device has failed 1 times in the last hour
[ 90.567937] EEH: Notify device drivers to shutdown
[ 90.567985] EEH: Collect temporary log
[ 90.568971] EEH: Reset without hotplug activity
[ 94.585540] EEH: Notify device drivers the completion of reset
[ 94.585934] EEH: Notify device driver to resume
----
The story about this bug is: Without commit 2aa5cf9 ("powerpc/eeh: Fix missed PE#0 on P7IOC"). PE#0 is regarded as invalid one. When kernel sees the frozen PE#0, the frozen state is cleared and dump the PHB diag-data, then try to recover it. When resetting the PE, the driver, which wasn't stopped by error_detected() completely, access the MMIO space and just causes another (recursive) EEH error. Eventually, the EEH recovery failed. During the PE reset, the I/O path for the PE should be frozen and MMIO access during the period should be dropped to avoid recursive EEH error.
Related branches
affects: | ubuntu → linux (Ubuntu) |
Changed in linux (Ubuntu): | |
assignee: | nobody → Taco Screen team (taco-screen-team) |
Changed in linux (Ubuntu Vivid): | |
status: | New → Fix Released |
assignee: | nobody → Tim Gardner (timg-tpi) |
status: | Fix Released → In Progress |
Changed in linux (Ubuntu Wily): | |
status: | New → Fix Released |
Changed in linux (Ubuntu Vivid): | |
status: | In Progress → Fix Committed |
tags: |
added: verification-done-vivid removed: verification-needed-vivid |
Default Comment by Bridge