lxd machine-resources CPU code fails to accurately calculate numa topology for AMD Opteron 6376

Bug #1869990 reported by Michael Mallon
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Medium
Alberto Donato
2.7
Fix Released
Medium
Alberto Donato

Bug Description

maas version: Snap package 2.7.0-8240-g.8d4770920 (current edge but was also happening on 2.7 stable snap)

I have a Dell C6145 with 4 sockets of AMD Opteron 6376 that I'm trying to commission. It fails to report any of the machine resources with the error commissioning script '50-maas-01-commissioning' failed during post-processing. This then causes the smartctl-validate tests to fail as maas doesn't detect any hardware.

I've traced the problem to the lxd resources code. The problem is that the code assumes that a cpu thread pair belong to the same numa node. In the case of the 6376, that isn't true. Here's the lscpu for that host:

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 4
NUMA node(s): 8
Vendor ID: AuthenticAMD
CPU family: 21
Model: 2
Model name: AMD Opteron(tm) Processor 6376
Stepping: 0
CPU MHz: 2580.498
BogoMIPS: 4600.10
Virtualization: AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache: 2048K
L3 cache: 6144K
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
NUMA node2 CPU(s): 16-23
NUMA node3 CPU(s): 24-31
NUMA node4 CPU(s): 32-39
NUMA node5 CPU(s): 40-47
NUMA node6 CPU(s): 48-55
NUMA node7 CPU(s): 56-63

Sockets are two numa nodes (eg. node0 and node1). Cores have a thread in each node (ie. thread 0 and thread 8 report the same core_id). It looks like it traverses the /sys/devices/system/cpu tree alphabetically and therefore hides numa nodes. With 4 sockets, only numa nodes 0, 1, 2, 4 and 6 are reported.

Then when https://github.com/maas/maas/blob/2.7/src/metadataserver/builtin_scripts/hooks.py#L453 processes the memory topology (which reports all 8 numa nodes correctly), node 3 doesn't exist in the numa_nodes dictionary and throws a key error at line 550/551:

        Traceback (most recent call last):
          File "/snap/maas/5328/lib/python3.6/site-packages/metadataserver/api.py", line 802, in signal
            target_status = process(node, request, status)
          File "/snap/maas/5328/lib/python3.6/site-packages/metadataserver/api.py", line 622, in _process_commissioning
            node, node.current_commissioning_script_set, request, status
          File "/snap/maas/5328/lib/python3.6/site-packages/metadataserver/api.py", line 515, in _store_results
            **args, timedout=(status == SIGNAL_STATUS.TIMEDOUT)
          File "/snap/maas/5328/lib/python3.6/site-packages/metadataserver/models/scriptresult.py", line 388, in store_result
            exit_status=self.exit_status,
        --- <exception caught here> ---
          File "/snap/maas/5328/lib/python3.6/site-packages/metadataserver/api.py", line 441, in try_or_log_event
            func(*args, **kwargs)
          File "/snap/maas/5328/lib/python3.6/site-packages/metadataserver/builtin_scripts/hooks.py", line 453, in process_lxd_results
            node.memory, numa_nodes = _parse_memory(data, numa_nodes)
          File "/snap/maas/5328/lib/python3.6/site-packages/metadataserver/builtin_scripts/hooks.py", line 551, in _parse_memory
            memory_node["total"] / 1024 / 1024
        builtins.KeyError: 3

Revision history for this message
Michael Mallon (durinix) wrote :
description: updated
Revision history for this message
Alberto Donato (ack) wrote :

Hi, could you please provide the output of the following command from the machine?

ls -l /sys/devices/system/cpu/*/node*

Changed in maas:
status: New → Incomplete
Revision history for this message
Michael Mallon (durinix) wrote :
Download full text (5.9 KiB)

lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu0/node0 -> ../../node/node0
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu10/node1 -> ../../node/node1
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu11/node1 -> ../../node/node1
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu12/node1 -> ../../node/node1
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu13/node1 -> ../../node/node1
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu14/node1 -> ../../node/node1
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu15/node1 -> ../../node/node1
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu16/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu17/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu18/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu19/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu1/node0 -> ../../node/node0
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu20/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu21/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu22/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu23/node2 -> ../../node/node2
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu24/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu25/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu26/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu27/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu28/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu29/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu2/node0 -> ../../node/node0
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu30/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu31/node3 -> ../../node/node3
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu32/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu33/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu34/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu35/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu36/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu37/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu38/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 30 05:36 /sys/devices/system/cpu/cpu39/node4 -> ../../node/node4
lrwxrwxrwx 1 root root 0 Mar 31 23:54 /sys/devices/system/cpu/cpu3/node0 -> ../../node/node0
lrwxrwxr...

Read more...

Revision history for this message
Alberto Donato (ack) wrote :

Thanks!

One last bit, can you also paste the output of

for f in /sys/devices/system/cpu/*/topology; do echo $f $(< $f/physical_package_id) $(< $f/core_id); done

Revision history for this message
Michael Mallon (durinix) wrote :

/sys/devices/system/cpu/cpu0/topology 0 0
/sys/devices/system/cpu/cpu10/topology 0 2
/sys/devices/system/cpu/cpu11/topology 0 3
/sys/devices/system/cpu/cpu12/topology 0 4
/sys/devices/system/cpu/cpu13/topology 0 5
/sys/devices/system/cpu/cpu14/topology 0 6
/sys/devices/system/cpu/cpu15/topology 0 7
/sys/devices/system/cpu/cpu16/topology 1 0
/sys/devices/system/cpu/cpu17/topology 1 1
/sys/devices/system/cpu/cpu18/topology 1 2
/sys/devices/system/cpu/cpu19/topology 1 3
/sys/devices/system/cpu/cpu1/topology 0 1
/sys/devices/system/cpu/cpu20/topology 1 4
/sys/devices/system/cpu/cpu21/topology 1 5
/sys/devices/system/cpu/cpu22/topology 1 6
/sys/devices/system/cpu/cpu23/topology 1 7
/sys/devices/system/cpu/cpu24/topology 1 0
/sys/devices/system/cpu/cpu25/topology 1 1
/sys/devices/system/cpu/cpu26/topology 1 2
/sys/devices/system/cpu/cpu27/topology 1 3
/sys/devices/system/cpu/cpu28/topology 1 4
/sys/devices/system/cpu/cpu29/topology 1 5
/sys/devices/system/cpu/cpu2/topology 0 2
/sys/devices/system/cpu/cpu30/topology 1 6
/sys/devices/system/cpu/cpu31/topology 1 7
/sys/devices/system/cpu/cpu32/topology 2 0
/sys/devices/system/cpu/cpu33/topology 2 1
/sys/devices/system/cpu/cpu34/topology 2 2
/sys/devices/system/cpu/cpu35/topology 2 3
/sys/devices/system/cpu/cpu36/topology 2 4
/sys/devices/system/cpu/cpu37/topology 2 5
/sys/devices/system/cpu/cpu38/topology 2 6
/sys/devices/system/cpu/cpu39/topology 2 7
/sys/devices/system/cpu/cpu3/topology 0 3
/sys/devices/system/cpu/cpu40/topology 2 0
/sys/devices/system/cpu/cpu41/topology 2 1
/sys/devices/system/cpu/cpu42/topology 2 2
/sys/devices/system/cpu/cpu43/topology 2 3
/sys/devices/system/cpu/cpu44/topology 2 4
/sys/devices/system/cpu/cpu45/topology 2 5
/sys/devices/system/cpu/cpu46/topology 2 6
/sys/devices/system/cpu/cpu47/topology 2 7
/sys/devices/system/cpu/cpu48/topology 3 0
/sys/devices/system/cpu/cpu49/topology 3 1
/sys/devices/system/cpu/cpu4/topology 0 4
/sys/devices/system/cpu/cpu50/topology 3 2
/sys/devices/system/cpu/cpu51/topology 3 3
/sys/devices/system/cpu/cpu52/topology 3 4
/sys/devices/system/cpu/cpu53/topology 3 5
/sys/devices/system/cpu/cpu54/topology 3 6
/sys/devices/system/cpu/cpu55/topology 3 7
/sys/devices/system/cpu/cpu56/topology 3 0
/sys/devices/system/cpu/cpu57/topology 3 1
/sys/devices/system/cpu/cpu58/topology 3 2
/sys/devices/system/cpu/cpu59/topology 3 3
/sys/devices/system/cpu/cpu5/topology 0 5
/sys/devices/system/cpu/cpu60/topology 3 4
/sys/devices/system/cpu/cpu61/topology 3 5
/sys/devices/system/cpu/cpu62/topology 3 6
/sys/devices/system/cpu/cpu63/topology 3 7
/sys/devices/system/cpu/cpu6/topology 0 6
/sys/devices/system/cpu/cpu7/topology 0 7
/sys/devices/system/cpu/cpu8/topology 0 0
/sys/devices/system/cpu/cpu9/topology 0 1

Alberto Donato (ack)
Changed in maas:
status: Incomplete → In Progress
assignee: nobody → Alberto Donato (ack)
importance: Undecided → Medium
milestone: none → 2.8.0b1
Revision history for this message
Alberto Donato (ack) wrote :
Revision history for this message
Alberto Donato (ack) wrote :

could you please download the following binary

https://dl.stgraber.org/lxd-resources

run it (under sudo) on the machine and attach the output?

I't a build from the LXD branch with the fix (https://github.com/lxc/lxd/pull/7118),
and should confirm that numa nodes are correctly reported per-thread.

Revision history for this message
Michael Mallon (durinix) wrote :
Download full text (27.9 KiB)

cpu:
  architecture: x86_64
  sockets:
  - name: AMD Opteron(tm) Processor 6376
    vendor: AuthenticAMD
    socket: 0
    cache:
    - level: 1
      type: Data
      size: 16384
    - level: 1
      type: Instruction
      size: 65536
    - level: 2
      type: Unified
      size: 2097152
    - level: 3
      type: Unified
      size: 6291456
    cores:
    - core: 0
      threads:
      - id: 0
        numa_node: 0
        thread: 0
        online: true
      - id: 8
        numa_node: 1
        thread: 1
        online: true
    - core: 1
      threads:
      - id: 1
        numa_node: 0
        thread: 0
        online: true
      - id: 9
        numa_node: 1
        thread: 1
        online: true
    - core: 2
      threads:
      - id: 10
        numa_node: 1
        thread: 0
        online: true
      - id: 2
        numa_node: 0
        thread: 1
        online: true
    - core: 3
      threads:
      - id: 11
        numa_node: 1
        thread: 0
        online: true
      - id: 3
        numa_node: 0
        thread: 1
        online: true
    - core: 4
      threads:
      - id: 12
        numa_node: 1
        thread: 0
        online: true
      - id: 4
        numa_node: 0
        thread: 1
        online: true
    - core: 5
      threads:
      - id: 13
        numa_node: 1
        thread: 0
        online: true
      - id: 5
        numa_node: 0
        thread: 1
        online: true
    - core: 6
      threads:
      - id: 14
        numa_node: 1
        thread: 0
        online: true
      - id: 6
        numa_node: 0
        thread: 1
        online: true
    - core: 7
      threads:
      - id: 15
        numa_node: 1
        thread: 0
        online: true
      - id: 7
        numa_node: 0
        thread: 1
        online: true
  - name: AMD Opteron(tm) Processor 6376
    vendor: AuthenticAMD
    socket: 1
    cache:
    - level: 1
      type: Data
      size: 16384
    - level: 1
      type: Instruction
      size: 65536
    - level: 2
      type: Unified
      size: 2097152
    - level: 3
      type: Unified
      size: 6291456
    cores:
    - core: 0
      threads:
      - id: 16
        numa_node: 2
        thread: 0
        online: true
      - id: 24
        numa_node: 3
        thread: 1
        online: true
    - core: 1
      threads:
      - id: 17
        numa_node: 2
        thread: 0
        online: true
      - id: 25
        numa_node: 3
        thread: 1
        online: true
    - core: 2
      threads:
      - id: 18
        numa_node: 2
        thread: 0
        online: true
      - id: 26
        numa_node: 3
        thread: 1
        online: true
    - core: 3
      threads:
      - id: 19
        numa_node: 2
        thread: 0
        online: true
      - id: 27
        numa_node: 3
        thread: 1
        online: true
    - core: 4
      threads:
      - id: 20
        numa_node: 2
        thread: 0
        online: true
      - id: 28
        numa_node: 3
        thread: 1
        online: true
    - core: 5
      threads:
      - id: 21
        numa_node: 2
        thread: 0
        online: true
      - id: 29
        numa_node: 3
        thread: 1
        online: true
    - core: 6
      threads...

Revision history for this message
Stéphane Graber (stgraber) wrote :

Could you show:
 - lstopo (from hwloc package)
 - grep "" /sys/devices/system/node/node*/distance

That would help us understand how this weird setup came to be :)

Revision history for this message
Stéphane Graber (stgraber) wrote :

And could you run this newer lxd-resources binary too?

https://dl.stgraber.org/lxd-resources

I've now got it CPU die aware which should give a much better topology on your system.

Revision history for this message
Stéphane Graber (stgraber) wrote :

https://github.com/lxc/lxd/pull/7120 is the follow-up upstream PR but we have no systems with die_id reporting anything other than 0 (or missing or -1), so your output would be quite useful.

It's also worth noting that "lscpu" was most likely telling you lies as it's unaware of CPU dies and instead was confusingly reporting threads. We'll see if our output matches what we'd expect given that platform.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Oh and worth noting that you'll need a recent kernel for die_id to be a thing.
Looks like 4.15 doesn't have that information while 5.3 and possibly something a bit earlier than that does.

Revision history for this message
Michael Mallon (durinix) wrote :
Download full text (6.2 KiB)

lstopo:

Machine (244GB total)
  Package L#0
    NUMANode L#0 (P#0 31GB)
      L3 L#0 (6144KB)
        L2 L#0 (2048KB) + L1i L#0 (64KB)
          L1d L#0 (16KB) + Core L#0 + PU L#0 (P#0)
          L1d L#1 (16KB) + Core L#1 + PU L#1 (P#1)
        L2 L#1 (2048KB) + L1i L#1 (64KB)
          L1d L#2 (16KB) + Core L#2 + PU L#2 (P#2)
          L1d L#3 (16KB) + Core L#3 + PU L#3 (P#3)
        L2 L#2 (2048KB) + L1i L#2 (64KB)
          L1d L#4 (16KB) + Core L#4 + PU L#4 (P#4)
          L1d L#5 (16KB) + Core L#5 + PU L#5 (P#5)
        L2 L#3 (2048KB) + L1i L#3 (64KB)
          L1d L#6 (16KB) + Core L#6 + PU L#6 (P#6)
          L1d L#7 (16KB) + Core L#7 + PU L#7 (P#7)
      HostBridge L#0
        PCIBridge
          PCI 1000:005b
            Block(Disk) L#0 "sda"
        PCIBridge
          PCI 8086:10c9
            Net L#1 "eno1"
          PCI 8086:10c9
            Net L#2 "eno2"
        PCI 1002:4390
        PCI 1002:439c
        PCIBridge
          PCI 1a03:2000
            GPU L#3 "controlD64"
            GPU L#4 "card0"
    NUMANode L#1 (P#1 31GB) + L3 L#1 (6144KB)
      L2 L#4 (2048KB) + L1i L#4 (64KB)
        L1d L#8 (16KB) + Core L#8 + PU L#8 (P#8)
        L1d L#9 (16KB) + Core L#9 + PU L#9 (P#9)
      L2 L#5 (2048KB) + L1i L#5 (64KB)
        L1d L#10 (16KB) + Core L#10 + PU L#10 (P#10)
        L1d L#11 (16KB) + Core L#11 + PU L#11 (P#11)
      L2 L#6 (2048KB) + L1i L#6 (64KB)
        L1d L#12 (16KB) + Core L#12 + PU L#12 (P#12)
        L1d L#13 (16KB) + Core L#13 + PU L#13 (P#13)
      L2 L#7 (2048KB) + L1i L#7 (64KB)
        L1d L#14 (16KB) + Core L#14 + PU L#14 (P#14)
        L1d L#15 (16KB) + Core L#15 + PU L#15 (P#15)
  Package L#1
    NUMANode L#2 (P#2 31GB) + L3 L#2 (6144KB)
      L2 L#8 (2048KB) + L1i L#8 (64KB)
        L1d L#16 (16KB) + Core L#16 + PU L#16 (P#16)
        L1d L#17 (16KB) + Core L#17 + PU L#17 (P#17)
      L2 L#9 (2048KB) + L1i L#9 (64KB)
        L1d L#18 (16KB) + Core L#18 + PU L#18 (P#18)
        L1d L#19 (16KB) + Core L#19 + PU L#19 (P#19)
      L2 L#10 (2048KB) + L1i L#10 (64KB)
        L1d L#20 (16KB) + Core L#20 + PU L#20 (P#20)
        L1d L#21 (16KB) + Core L#21 + PU L#21 (P#21)
      L2 L#11 (2048KB) + L1i L#11 (64KB)
        L1d L#22 (16KB) + Core L#22 + PU L#22 (P#22)
        L1d L#23 (16KB) + Core L#23 + PU L#23 (P#23)
    NUMANode L#3 (P#3 31GB) + L3 L#3 (6144KB)
      L2 L#12 (2048KB) + L1i L#12 (64KB)
        L1d L#24 (16KB) + Core L#24 + PU L#24 (P#24)
        L1d L#25 (16KB) + Core L#25 + PU L#25 (P#25)
      L2 L#13 (2048KB) + L1i L#13 (64KB)
        L1d L#26 (16KB) + Core L#26 + PU L#26 (P#26)
        L1d L#27 (16KB) + Core L#27 + PU L#27 (P#27)
      L2 L#14 (2048KB) + L1i L#14 (64KB)
        L1d L#28 (16KB) + Core L#28 + PU L#28 (P#28)
        L1d L#29 (16KB) + Core L#29 + PU L#29 (P#29)
      L2 L#15 (2048KB) + L1i L#15 (64KB)
        L1d L#30 (16KB) + Core L#30 + PU L#30 (P#30)
        L1d L#31 (16KB) + Core L#31 + PU L#31 (P#31)
  Package L#2
    NUMANode L#4 (P#4 24GB) + L3 L#4 (6144KB)
      L2 L#16 (2048KB) + L1i L#16 (64KB)
        L1d L#32 (16KB) + Core L#32 + PU L#32 (P#32)
        L1d L#33 (16KB) + Core L#33 + PU L#33 (P#33)
      L2 L#17 (2048KB) + L1i L#17 (64KB)
        L1d L#34 (16KB) +...

Read more...

Revision history for this message
Stéphane Graber (stgraber) wrote :

Could you also show /proc/cpuinfo for good measure?

Given that this platform doesn't have nor sets die_id, I wonder if there's something else in there that could be used to make a unique combination.

Revision history for this message
Michael Mallon (durinix) wrote :
Download full text (28.3 KiB)

cpu:
  architecture: x86_64
  sockets:
  - name: AMD Opteron(tm) Processor 6376
    vendor: AuthenticAMD
    socket: 0
    cache:
    - level: 1
      type: Data
      size: 16384
    - level: 1
      type: Instruction
      size: 65536
    - level: 2
      type: Unified
      size: 2097152
    - level: 3
      type: Unified
      size: 6291456
    cores:
    - core: 0
      die: 0
      threads:
      - id: 0
        numa_node: 0
        thread: 0
        online: true
      - id: 8
        numa_node: 1
        thread: 1
        online: true
    - core: 1
      die: 0
      threads:
      - id: 1
        numa_node: 0
        thread: 0
        online: true
      - id: 9
        numa_node: 1
        thread: 1
        online: true
    - core: 2
      die: 0
      threads:
      - id: 10
        numa_node: 1
        thread: 0
        online: true
      - id: 2
        numa_node: 0
        thread: 1
        online: true
    - core: 3
      die: 0
      threads:
      - id: 11
        numa_node: 1
        thread: 0
        online: true
      - id: 3
        numa_node: 0
        thread: 1
        online: true
    - core: 4
      die: 0
      threads:
      - id: 12
        numa_node: 1
        thread: 0
        online: true
      - id: 4
        numa_node: 0
        thread: 1
        online: true
    - core: 5
      die: 0
      threads:
      - id: 13
        numa_node: 1
        thread: 0
        online: true
      - id: 5
        numa_node: 0
        thread: 1
        online: true
    - core: 6
      die: 0
      threads:
      - id: 14
        numa_node: 1
        thread: 0
        online: true
      - id: 6
        numa_node: 0
        thread: 1
        online: true
    - core: 7
      die: 0
      threads:
      - id: 15
        numa_node: 1
        thread: 0
        online: true
      - id: 7
        numa_node: 0
        thread: 1
        online: true
  - name: AMD Opteron(tm) Processor 6376
    vendor: AuthenticAMD
    socket: 1
    cache:
    - level: 1
      type: Data
      size: 16384
    - level: 1
      type: Instruction
      size: 65536
    - level: 2
      type: Unified
      size: 2097152
    - level: 3
      type: Unified
      size: 6291456
    cores:
    - core: 0
      die: 0
      threads:
      - id: 16
        numa_node: 2
        thread: 0
        online: true
      - id: 24
        numa_node: 3
        thread: 1
        online: true
    - core: 1
      die: 0
      threads:
      - id: 17
        numa_node: 2
        thread: 0
        online: true
      - id: 25
        numa_node: 3
        thread: 1
        online: true
    - core: 2
      die: 0
      threads:
      - id: 18
        numa_node: 2
        thread: 0
        online: true
      - id: 26
        numa_node: 3
        thread: 1
        online: true
    - core: 3
      die: 0
      threads:
      - id: 19
        numa_node: 2
        thread: 0
        online: true
      - id: 27
        numa_node: 3
        thread: 1
        online: true
    - core: 4
      die: 0
      threads:
      - id: 20
        numa_node: 2
        thread: 0
        online: true
      - id: 28
        numa_node: 3
        thread: 1
        online: true
    - core: 5
      die: 0
      thre...

Revision history for this message
Michael Mallon (durinix) wrote :
Download full text (76.7 KiB)

processor : 0
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 6376
stepping : 0
microcode : 0x6000852
cpu MHz : 1178.907
cache size : 2048 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
apicid : 32
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 4600.21
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor : 1
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 6376
stepping : 0
microcode : 0x6000852
cpu MHz : 753.207
cache size : 2048 KB
physical id : 0
siblings : 16
core id : 1
cpu cores : 8
apicid : 33
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 4600.21
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

processor : 2
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 6376
stepping : 0
microcode : 0x6000852
cpu MHz : 537.606
cache size : 2048 KB
physical id : 0
siblings : 16
core id : 2
cpu cores : 8
apicid : 34
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tc...

Revision history for this message
Michael Mallon (durinix) wrote :
Download full text (7.8 KiB)

What if instead of visiting each cpu individually, you visit by numa node and only consider thread grouping inside a numa node? Then you could go back to the old datastructure. You could also fall back to visiting by cpu directly if the node interface doesn't exist:

user@host:~# ls -salh /sys/devices/system/node/node*/cpu*
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu0 -> ../../cpu/cpu0
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu1 -> ../../cpu/cpu1
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu2 -> ../../cpu/cpu2
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu3 -> ../../cpu/cpu3
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu4 -> ../../cpu/cpu4
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu5 -> ../../cpu/cpu5
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu6 -> ../../cpu/cpu6
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node0/cpu7 -> ../../cpu/cpu7
0 -r--r--r-- 1 root root 4.0K Apr 2 01:10 /sys/devices/system/node/node0/cpulist
0 -r--r--r-- 1 root root 4.0K Apr 2 00:42 /sys/devices/system/node/node0/cpumap
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu10 -> ../../cpu/cpu10
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu11 -> ../../cpu/cpu11
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu12 -> ../../cpu/cpu12
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu13 -> ../../cpu/cpu13
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu14 -> ../../cpu/cpu14
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu15 -> ../../cpu/cpu15
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu8 -> ../../cpu/cpu8
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node1/cpu9 -> ../../cpu/cpu9
0 -r--r--r-- 1 root root 4.0K Apr 2 01:10 /sys/devices/system/node/node1/cpulist
0 -r--r--r-- 1 root root 4.0K Apr 2 00:42 /sys/devices/system/node/node1/cpumap
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu16 -> ../../cpu/cpu16
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu17 -> ../../cpu/cpu17
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu18 -> ../../cpu/cpu18
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu19 -> ../../cpu/cpu19
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu20 -> ../../cpu/cpu20
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu21 -> ../../cpu/cpu21
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu22 -> ../../cpu/cpu22
0 lrwxrwxrwx 1 root root 0 Apr 2 01:10 /sys/devices/system/node/node2/cpu23 -> ../../cpu/cpu23
0 -r--r--r-- 1 root root 4.0K Apr 2 01:10 /sys/devices/system/node/node2/cpulist
0 -r--r--r-- 1 root root 4.0K Apr 2 00:42 /sys/devices/system/node/node2/cpumap
0 lrwxrwxrwx 1 root root 0...

Read more...

Revision history for this message
Michael Mallon (durinix) wrote :

Please let me know if you need any more tests or data from me.

Alberto Donato (ack)
Changed in maas:
status: In Progress → Fix Committed
Alberto Donato (ack)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.