Broadwell ECRC Support missing in Ubuntu

[Mark W Wenning]

Here is the problem statement from the Dell team:

When booting into Ubuntu 14.04.4 with a Broadwell CPU and an Intel Quick Assist Card, the memory location that corresponds to ECRC is set to 0x01e0, when the BIOS is setting this location 0x00a0 pre-OS boot. This causes the card to not function unless we implement the following workaround using setpci.

“setpci –s AA:BB.C 160.w=0”, where AA:BB.C is the PCI Root Path for the Intel Quick Assit Card.

We’ve verified the memory location is correct when booting to other OSes, such as RHEL 7.2 and Windows Server 2012 R2.

If there is any information you can give as to why this may be occurring in Ubuntu or where we may start to debug when the memory is changed, we would appreciate it.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1571798

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Hi Mark,

Do we know if this is a regression? It could be since RHEL 7.2 does not have the bug. Does this issue happen in 14.04.3 or 14.04.2? If it is a regression, we can go the kernel bisect route and try to find the commit that introduced this.

It would also be good if they can test the latest mainline kernel to see if this bug is already fixed there. It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc4-wily/

Thanks in advance!

[John Mazzie (john-mazzie)]

Attached are the requested logs from the system with the current issue. I was not able to run apport due to our proxy.

The card in question is current at PCI Root Path 80:02.0 (Intel Quick Assist 8950).

When running the command "setpci -s 80:02.0 160.w", the value returned is "0000:80:02.0 @160 = 01e0", but the expectation is that it will be "0000:80:02.0 @160 = 00a0", with the BIOS version we are running.

The issue also appeared on the 16.04 beta release that I tried.

[Mark W Wenning (mwenning)]

Thanks John,

Can you test some earlier versions of Ubuntu 14.04 and see if you have the same problem?

[John Mazzie (john-mazzie)]

Mark,

I just finished testing with 14.04.1 and the value is correct. The value returned is "0000:80:02.0 @160 = 00a0". I'm going to try with 14.04.2 and 14.04.3 as well to see where the change occurs. I've also attached the "apport-cli linux" logs from my 14.04.1 test.

[Mark W Wenning (mwenning)]

Fantastic, thanks John.

[John Mazzie (john-mazzie)]

I've just completed the test with 14.04.2 and 14.04.3. It appears to be correct in 14.04.2 (00a0) but broken in 14.04.3 (01e0). I'm attaching the apport logs from both tests. 3.16.0-30 corresponds to 14.04.2 and 3.19.0-25 corresponds to 14.04.3.

Thanks,
John

[John Mazzie (john-mazzie)]

I was just curious if there was anything else that I could provide to help with the debugging of this issue. Please let me know.

Thanks,
John Mazzie

[Joseph Salisbury (jsalisbury)]

Did you happen to have a chance to test the latest mainline kernel, posted in comment #2? Mainline is now up to -rc7, so the link to the kernel would be:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc7-wily/

If the bug is fixed in mainline, we can perform a reverse bisect to identify the commit that fixes it.

However, if the bug still exists in mainline, we would want to perform a regular kernel bisect to find the commit that introduced the regression. To perform a bisect, we need to identify the last good and first bad kernel versions. Per comment #7 v3.16 is good, but v3.19 is bad. We would want to test the versions in between. So if mainline still has the bug, can you test the following upstream kernels:

v3.17: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-utopic/
v3.18: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-vivid/

Once we have these results, we can narrow down the versions further to individual release candidates.

Thanks in advance!

[John Mazzie (john-mazzie)]

Here are the results from upgrading the mainline kernel.

Started at 14.04.4 and upgraded to kernel 4.6 - Error is still occurring.

Started at 14.04.2 and upgraded to kernel 3.17 - Still working as expected
Upgraded from kernel 3.17 to 3.18 - Error starts to appear.

It looks like something happened between kernel version 3.17 and 3.18.

Thanks,
John Mazzie

[Joseph Salisbury (jsalisbury)]

Thanks for testing, John. To narrow it down further, can you test some of the 3.18 release candidates?

They can all be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/

I'd suggest trying -rc1 first. If that version does not have the bug, try some of the newer candidates:

v3.18-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-rc1-utopic/
v3.18-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-rc4-vivid/
v3.18-rc7: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-rc7-vivid/

Once we know the last good kernel version and the first bad one, we can bisect between them.

[John Mazzie (john-mazzie)]

It looks like the break happened in the initial release candidate. 3.18rc1

[Joseph Salisbury (jsalisbury)]

I started a kernel bisect between v3.17 final and v3.18-rc1. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
35a9ad8af0bb0fa3525e6d0d20e32551d226f38e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

The test Kernel provided does not cause the error.

[Joseph Salisbury (jsalisbury)]

I built the first test kernel, up to the following commit:
d6dd50e07c5bec00db2005969b1a01f8ca3d25ef

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

Error occurred after this update.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
bf65dea87e87c53ba4f97c6432761498bc977efd

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

The error still exists in this kernel.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
b211e9d7c861bdb37b86d6384da9edfb80949ceb

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

Error still exists in this update.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
80213c03c4151d900cf293ef0fc51f8d88495e14

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

The error is still present.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
ea584595fc85e65796335033dfca25ed655cd0ed

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

The error is corrected in this build.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
2828c9cdb8bd30f49c48210c014ccdd4cb994931

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Hide

[John Mazzie (john-mazzie)]

I had issues installing this set of kernel files. I'm receiving the following errors when installing the *.deb package.

Errors were encountered while processing: linux-headers-3.17.0-031700rc1-generic.

I downloaded the mainline linux-headers-3.17.0-031700rc1_3.17.0-031700rc1.201409021903_all.deb file since this package was named 3.17.0-031700rc1, which let me install the updated kernel with the required dependencies.

The error seems to be fixed in this version as well.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
7b31997a734cd24c305d5c58a366e4c8f7673e02

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is also working correctly.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
867f667fb9c6734e06cc24e96fc7f06a7e772084

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is working correctly.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
a092e19b688be88f7329bd05f90cb92ebe1a4f5b

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is also working.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
3631073659d0aafeaa52227bb61a100efaf901dc

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is working as well.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
afa3536be88b435a057cb727b48fd3d760a497d2

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is also correct.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
782d59c5dfc5ac39ac8cfb4c6dd40597938dde9c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is working correctly.

[Joseph Salisbury (jsalisbury)]

ea584595fc85e65796335033dfca25ed655cd0ed was reported as the first bad commit, but it is a merge:

commit ea584595fc85e65796335033dfca25ed655cd0ed
Merge: 782d59c a092e19
Author: Linus Torvalds <email address hidden>
Date: Thu Oct 9 14:58:15 2014 -0400

    Merge tag 'gpio-v3.18-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-gpio

The bisect may have gone bad due to marking a test kernel good when it was actually bad or vice versa. We may need to test some of the previously posted kernels and ensure they were really bad or good.

[John Mazzie (john-mazzie)]

I will go back and retest the kernels you've posted. I'll start with the last one I said was bad and verify that it was actually bad, then go forward.

[Joseph Salisbury (jsalisbury)]

Thanks, John. I copied all the test kernels back to:
http://kernel.ubuntu.com/~jsalisbury/lp1571798/

Each test kernel is under the corresponding SHA1.

[John Mazzie (john-mazzie)]

I rechecked and it turns out I misread commit ea584595fc85e65796335033dfca25ed655cd0ed. It appears to be working in that build as well.

[John Mazzie (john-mazzie)]

It looks like it may have been poor wording on my part. I said that "The error is corrected in this build." I should have left it simply that it was working in that build.

[Joseph Salisbury (jsalisbury)]

Thanks for re-running the test, John. I restarted the bisect and marked ea584595fc85e65796335033dfca25ed655cd0ed as good this time.

I built the next test kernel, up to the following commit:
783a28ec0bf2c2d560d8004c92919d112a777e55

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This kernel version is broken.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
134cd00d766fc7014b53d9cea67a6bcb894ae51e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is working correctly.

[Joseph Salisbury (jsalisbury)]

I listed the SHA1 in comment #46 incorrectly. It was actually: 7e8dd58bd1a6cebb4c398ae6550cad9b44c3d45c

The next test kernel is now ready, and it is actually built up to:
134cd00d766fc7014b53d9cea67a6bcb894ae51e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

[John Mazzie (john-mazzie)]

This version is working correctly.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
6a73336bde293741026614135419e9b76afb9145

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is also working correctly.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
cf8d7b589c53f17e10e9f1ef91dd9e2ba3ca9a7c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is broken.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
81ee57326c9ca612436bd6c98258942d57063c98

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is working.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
302328c00341f1c161bfe32d085d3e6549a08f2d

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is working.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
d537a3abb4b7085ebc3ce35e64acbad8ece1eece

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is broken.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
1302fcf0d03e6ea74846c7fee14736306ab2ce4b

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[John Mazzie (john-mazzie)]

This version is broken.

[Joseph Salisbury (jsalisbury)]

The bisect reported the following as the first bad commit:

commit 1302fcf0d03e6ea74846c7fee14736306ab2ce4b
Author: Bjorn Helgaas <email address hidden>
Date: Sat Aug 30 07:23:01 2014 -0600

    PCI: Configure *all* devices, not just hot-added ones

I built a Xenial test kernel with a revert of this commit.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1571798/revert

Can you test that kernel and report back if it has the bug or not?

Note, with this kernel, you have to install both the linux-image and linux-image-extra .deb packages.

[John Mazzie (john-mazzie)]

This version is working correctly.

[Joseph Salisbury (jsalisbury)]

I pinged upstream and the patch author requesting additional feedback.

[Bjorn Helgaas (bjorn-helgaas)]

The register in question is the Advanced Error Capabilities and Control register, at offset 0x18 in the Advanced Error Reporting capability, which starts at 0x148 in the config space of device 80:02.0.

In the pre-boot value of 0x00a0, the following bits are set (per PCIe spec r3.0, sec 7.10.7, these bits are read-only):

  PCI_ERR_CAP_ECRC_GENC 0x00000020 /* ECRC Generation Capable */
  PCI_ERR_CAP_ECRC_CHKC 0x00000080 /* ECRC Check Capable */

In the value of 0x01e0 after Linux boots, the following additional bits are set:

  PCI_ERR_CAP_ECRC_GENE 0x00000040 /* ECRC Generation Enable */
  PCI_ERR_CAP_ECRC_CHKE 0x00000100 /* ECRC Check Enable */

Linux is setting these bits in program_hpp_type2() because there is apparently an ACPI _HPX method that applies to this device, and it returns a PCI Express setting record (ACPI spec 5.0, sec 6.2.8.3) with an "Advanced Error Capabilities and Control Register OR Mask" that has PCI_ERR_CAP_ECRC_GENE and PCI_ERR_CAP_ECRC_CHKE set.

Can you collect an ACPI dump to confirm that this is the case?

As I mentioned in the 1302fcf0d03e changelog, it's not completely clear from the spec (ACPI 5.0, sec 6.2.8) when to apply these _HPX settings. It says OSPM should use them to "configure devices not configured by the platform firmware during initial system boot." The question is how OSPM can tell whether a device has been configured by platform firmware.

Since I don't know how to tell if a device has been configured by platform firmware, I chose to apply the _HPX settings to *all* devices.

Any BIOS folks want to suggest a way to tell whether firmware has configured a device?

[John Mazzie (john-mazzie)]

I've attached the ACPI Dump for the system with the card in it. The card is currently located on PCI Root Path 80:02.0, PCI Device Bus/Device/Function of 84:00.0.

[Mark W Wenning (mwenning)]

Bjorn Helgaas, have you looked at the ACPI dump? Any suggestions on how to proceed?

[Joseph Salisbury (jsalisbury)]

This bug has been stale for a bit. Are there any updates?

[Bjorn Helgaas (bjorn-helgaas)]

Please try the attached patch. It makes it so that if the device does not support ECRC generation or checking, we don't enable those features. Currently (without this patch), we *do* enable ECRC generation and checking if _HPX allows, i.e., if the platform can support ECRC.

The ACPI dump confirms my theory from comment #65 that the system supplies an _HPX method with PCI_ERR_CAP_ECRC_GENE and PCI_ERR_CAP_ECRC_CHKE set (see disassembly below).

Apparently the Intel Quick Assist card is at 85:00.0 ("Intel Corporation DH895XCC Series QAT"). Here's the path leading to it:

  pci 0000:80:02.0: [8086:6f04] # Xeon D PCI Express Root Port 2
  pci 0000:80:02.0: PCI bridge to [bus 83-86]
  pci 0000:83:00.0: [10b5:8724] # PLX 8724 Upstream Port
  pci 0000:83:00.0: PCI bridge to [bus 84-86]
  pci 0000:84:00.0: [10b5:8724] # PLX 8724 Downstream Port
  pci 0000:84:00.0: PCI bridge to [bus 85]
  pci 0000:85:00.0: [8086:0435] # DH895XCC Series QAT

Here are the ECRC settings along the path:

  80:02.0: AERCap: GenCap+ CGenEn+ ChkCap+ ChkEn+
  83:00.0: AERCap: GenCap+ CGenEn+ ChkCap+ ChkEn+
  84:00.0: AERCap: GenCap+ CGenEn+ ChkCap+ ChkEn+
  85:00.0: AERCap: GenCap- CGenEn+ ChkCap- ChkEn+

This looks suspect because 85:00.0 claims that it does not support ECRC Generation ("GenCap-") or ECRC Checking ("ChkCap-"), yet we set the Enable bits for both features. The workaround in the initial report turns off ECRC checking in 80:02.0. I suspect that turning off ECRC generation and checking in 85:00.0, e.g., "setpci -s85:00.0 118.w=0" would also be a workaround. This patch should be the equivalent of this setpci command.

Here's the _HPX disassembly from dsdt.dsl (extracted from comment #66):

        Device (PCI0)
            ...
            Method (_HPX, 0, NotSerialized) // _HPX: Hot Plug Parameter Extensions
            {
                Store ("_HPX", Debug)
                Name (SSDH, Package (0x01)
                {
                    Package (0x12)
                    {
                        0x02,
                        0x01,
                        0xFC000FCF, // Uncorrectable Mask AND
                        0x03A18000, // Uncorrectable Mask OR
                        0xFC000FCF, // Uncorrectable Severity AND
                        0x004E7030, // Uncorrectable Severity OR
                        0xFFFF0E3E, // Correctable Mask AND
                        0xF1C1, // Correctable Mask OR
                        0xFFFFFEBF, // AER AND
                        0x0140, // AER OR
                        0xFFF1, // Device Control AND
                        0x0E, // Device Control OR
                        0xFFFF, // Link Control AND
                        0x00, // Link Control OR
                        0xFFFFC010, // Secondary Uncorrectable Severity AND
                        0x1BC0, // Secondary Uncorrectable Severity OR
                        0xFFFFC010, // Secondary Uncorrectable Mask AND
                        0x242F // Secondary Uncorrectable Mask OR
                    }
                })
                Store (SSDH, Debug)
                Return (SSDH)
            }

[Joseph Salisbury (jsalisbury)]

I built a test kernel with the patch. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1571798

This test kernel needs both the linux-image and linux-image-extra .deb packages.

[Bjorn Helgaas (bjorn-helgaas)]

John, can you also include a little more detail about what the failure looks like to a user? Does this result in AER errors being logged? When we come up with a patch that works, I'd like to include a hint about what user-visible problems it fixes in the changelog. I see from the initial report that it "causes the card to not function", but I'd like to know if there are messages from AER or the driver that are connected to this problem.

[Andy Whitcroft (apw)]

This bug was nominated against a series that is no longer supported, ie yakkety. The bug task representing the yakkety nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.