© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
73416ba
(Get the code!)
BugLink: https:/
[Impact]
Intel Skylake server processors and onward have a different Last Level Cache (LLC) topology than earlier processors, and such processors have a new feature called Sub-NUMA-Clustering (SNC) which is similar to the existing Cluster-On-Die (CoD) feature earlier server processors has.
Sub-NUMA-Clustering divides the system into two "slices", each of which are allocated half the CPU cores, half the Last Level Cache and one memory controller. Each slice is enumerated as a NUMA node.
The difference between Sub-NUMA-Clustering and Cluster-On-Die is how the Last Level Cache is exposed to each NUMA node. CoD had the same cache line present in each half of the LLC. In SNC, each cache line is only present in its respective slice. Because of this, the semantics around accessing LLC changes, with a process accessing NUMA-local memory only seeing half the LLC capacity.
On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and Bionic 4.15 kernels we see the following oops during NUMA node enumeration:
.... node #0, CPUs: #1 #2 #3 #4 #5 #6
.... node #1, CPUs: #7
sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
WARNING: CPU: 7 PID: 0 at /build/
Modules linked in:
CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu
Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018
RIP: 0010:topology_
Call Trace:
set_cpu_
start_secondary
secondary_
#8 #9 #10 #11 #12 #13
.... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
.... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
smp: Brought up 2 nodes, 28 CPUs
This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10.
The oops happens because topology_sane() checks to see if the Last Level Cache line matches across different CPUs, which it no longer does.
[Fix]
The fix comes in the form of the following upstream commit, which landed in
Linux 4.17:
commit 1340ccfa9a9afef
Author: Alison Schofield <email address hidden>
Date: Fri Apr 6 17:21:30 2018 -0700
Subject: x86,sched: Allow topologies where NUMA nodes share an LLC
Link: https:/
The commit adds a check for this particular family of Intel processors, and if the CPU family matches, it simply skips the check to topology_sane().
The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with the only remarks being re-arranging #includes and small context fixups.
[Testcase]
Unfortunately, this is hardware specific. To test this, you need a Intel Skylake server processor which supports Sub-NUMA-
We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10 that has successfully tested the below test kernels, with good results.
Xenial 4.4 ppa:
https:/
Xenial 4.15 HWE ppa:
https:/
Running the test kernel, the oops does not reproduce:
smp: Bringing up secondary CPUs ...
x86: Booting SMP configuration:
.... node #0, CPUs: #1
NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
#2 #3 #4 #5 #6
.... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13
.... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
.... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
smp: Brought up 2 nodes, 28 CPUs
smpboot: Max logical packages: 1
smpboot: Total of 28 processors activated
[Regression Potential]
The commit modifies a small section of smpboot code, which every machine will execute on boot. The majority of the commit breaks up a large if statement into smaller blocks than it was previously, and adds an extra if statement to check for a specific processor family.
If a regression were to occur, some machines would or would not make their calls to topology_sane(), which in the worst case, would result in a oops message and slightly degraded performance. The system would still function normally.
The commit has been present since 4.17-rc2 and is present in Eoan and Focal. There are no fixup commits, and no additional processor families have been added since.
Because of the small re-arrangement in logic, and the addition of a processor family check, these changes a fairly minor, and I don't think it will cause any regressions.