Fix enabling bridge MMIO windows

Bug #1771344 reported by bugproxy on 2018-05-15
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
High
Canonical Kernel Team
linux (Ubuntu)
Status tracked in Cosmic
Artful
High
Joseph Salisbury
Bionic
High
Joseph Salisbury
Cosmic
High
Joseph Salisbury

Bug Description

== SRU Justification ==
IBM is requesting this patch in Bionic and Artful to fix a regression. The
regression was introduced in v3.11-rc1. The patch fixes enabling bridge
MMIO windows. Commit 13a83eac373c was also cc'd to upstream stable, and
has already landed in Xenial via upstream stable updates.

== Fix ==
13a83eac373c ("powerpc/eeh: Fix enabling bridge MMIO windows")

== Regression Potential ==
Low. Limited to powerpc and fixes a current regression.

== Test Case ==
A test kernel was built with this patch and tested by the original bug reporter.
The bug reporter states the test kernel resolved the bug.

== Comment: #0 - Breno Leitao <email address hidden>
On boot we save the configuration space of PCIe bridges. We do this so
when we get an EEH event and everything gets reset that we can restore
them.

Unfortunately we save this state before we've enabled the MMIO space
on the bridges. Hence if we have to reset the bridge when we come back
MMIO is not enabled and we end up taking an PE freeze when the driver
starts accessing again.

This patch forces the memory/MMIO and bus mastering on when restoring
bridges on EEH. Ideally we'd do this correctly by saving the
configuration space writes later, but that will have to come later in
a larger EEH rewrite. For now we have this simple fix.

The original bug can be triggered on a boston machine by doing:
 echo 0x8000000000000000 > /sys/kernel/debug/powerpc/PCI0001/err_injct_outbound
On boston, this PHB has a PCIe switch on it. Without this patch,
you'll see two EEH events, 1 expected and 1 the failure we are fixing
here. The second EEH event causes the anything under the PHB to
disappear (i.e. the i40e eth).

With this patch, only 1 EEH event occurs and devices properly recover.

This is commit id 13a83eac373c49c0a081cbcd137e79210fe78acd and should be part of Ubuntu 18.04 kernel.

bugproxy (bugproxy) on 2018-05-15
tags: added: architecture-ppc64le bugnameltc-167852 severity-high targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → linux (Ubuntu)
tags: added: triage-g
Changed in ubuntu-power-systems:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Manoj Iyer (manjo) on 2018-05-15
Changed in linux (Ubuntu):
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Kernel Team (canonical-kernel-team)
importance: Undecided → Critical
importance: Critical → High
status: New → Triaged
Changed in linux (Ubuntu Bionic):
importance: Undecided → High
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
assignee: Canonical Kernel Team (canonical-kernel-team) → Joseph Salisbury (jsalisbury)
status: Triaged → In Progress
Changed in linux (Ubuntu Bionic):
status: New → In Progress
Joseph Salisbury (jsalisbury) wrote :

I built a test kernel with commit 13a83eac373c49c0a081cbcd137e79210fe78acd. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1771344

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-image-unsigned, linux-modules and linux-modules-extra .deb packages.

Thanks in advance!

------- Comment From <email address hidden> 2018-05-22 14:42 EDT-------
(In reply to comment #4)
> I built a test kernel with commit 13a83eac373c49c0a081cbcd137e79210fe78acd.
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1771344
>
> Can you test this kernel and see if it resolves this bug?
>
> Note about installing test kernels:
> ? If the test kernel is prior to 4.15(Bionic) you need to install the
> linux-image and linux-image-extra .deb packages.
> ? If the test kernel is 4.15(Bionic) or newer, you need to install the
> linux-image-unsigned, linux-modules and linux-modules-extra .deb packages.
>
> Thanks in advance!

Thanks for the custom kernel. I tested it and it is working as expected:

? ~ uname -a
Linux ubuntu 4.15.0-20-generic #22~lp1771344 SMP Wed May 16 18:34:49 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Changed in ubuntu-power-systems:
status: Triaged → In Progress
description: updated
Changed in linux (Ubuntu Artful):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Artful):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Bionic):
status: In Progress → Invalid
Manoj Iyer (manjo) wrote :

I do not see this patch in bionic kernel, moving the status back to in progress.

Changed in linux (Ubuntu Bionic):
status: Invalid → In Progress
Khaled El Mously (kmously) wrote :

@Manoj, this patch looks to be in Bionic as 16735b38aeae1ce2ae21983a7a7440922d6941e4 - can you please double check?

Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-artful
Manoj Iyer (manjo) on 2018-06-18
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Released
Changed in ubuntu-power-systems:
status: In Progress → Fix Committed
Manoj Iyer (manjo) on 2018-06-18
Changed in linux (Ubuntu Cosmic):
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers