Comment 9 for bug 1832915

While I built a proper PPA in [1] this seems so trivial that we can rebuild locally with just

$ cc -std=gnu99 -I. -D__thread="" -c -o numad.o numad.c
$ cc numad.o -lpthread -lrt -lm -o numad
$ mv numad /usr/bin/numad

That should allow quick iterations.

With debug enabled I found that the second set is actually null.
So the red herring assumption above was correct.

996 while (nodes) {
997 if (ID_IS_IN_LIST(node_id, p->node_list_p)) {
998 OR_LISTS(cpu_bind_list_p, cpu_bind_list_p, node[node_id].cpu_list_p);
(gdb) p node[node_id].cpu_list_p
$4 = (id_list_p) 0x0

The arg is
(gdb) p *(p->node_list_p)
$7 = {set_p = 0x304a3a418d0, bytes = 8}

This delivers "1"
int nodes = NUM_IDS_IN_LIST(p->node_list_p);
(gdb) p nodes
$5 = 1

That is:
#define NUM_IDS_IN_LIST(list_p) CPU_COUNT_S(list_p->bytes, list_p->set_p)

Per [2] this counts the cpus in the cpu_set.

So the TL;DR of this loop
while (nodes) {
  nodes -= 1;
is that it iterates over all CPUs

On the each iteration it checks
  if (ID_IS_IN_LIST(node_id, p->node_list_p)) {

node_id starts at zero and is incremented each iteration.

I must admit the usage of the term "node" for cpus here is very misleading.

"node" is a global data structure

typedef struct node_data {
    uint64_t node_id;
    uint64_t MBs_total;
    uint64_t MBs_free;
    uint64_t CPUs_total; // scaled * ONE_HUNDRED
    uint64_t CPUs_free; // scaled * ONE_HUNDRED
    uint64_t magnitude; // hack: MBs * CPUs
    uint8_t *distance;
    id_list_p cpu_list_p;
} node_data_t, *node_data_p;
node_data_p node = NULL;

Due to the misperception of "node" actually being CPUs the indexing here is off IMHO.

(gdb) p node[0]
$13 = {node_id = 0, MBs_total = 65266, MBs_free = 1510, CPUs_total = 2000, CPUs_free = 1144, magnitude = 1727440, distance = 0x304a3a41850 "\n(\032\n\244~", cpu_list_p = 0x304a3a41810}
(gdb) p node[1]
$14 = {node_id = 8, MBs_total = 65337, MBs_free = 1734, CPUs_total = 2000, CPUs_free = 1049, magnitude = 1818966, distance = 0x304a3a418b0 "(\n\032\n\244~", cpu_list_p = 0x304a3a41870}

My CPUs are 0,4,8,... and so is the indexing here as despite the node name it is actually based on CPUs.

Summary:
- The code checks for each CPU as counted by NUM_IDS_IN_LIST
- It will increase the ID until it found a hit in ID_IS_IN_LIST(node_id, p->node_list_p)
- that will skip empty CPUs as in my SMT case
- Once it found a cpu that is in the set it will OR_LISTS
  node[node_id].cpu_list_p