Comment 0 for bug 1358835

== Comment: #0 - TSAI-YANG JEA <email address hidden> - 2014-08-12 14:52:38 ==
---Problem Description---

In a ppc64le P8 node with SMT 8 mode, numa_node_of_cpu (80) returns following warning:
libnuma: Warning: /sys not mounted or invalid. Assuming one node: No such file or directory

---uname output---
Linux c656f2n01 3.13.0-30-generic #55-Ubuntu SMP Fri Jul 4 21:29:58 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = Tuletta P8 22L

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 Problem can be created with following test case on a P8 LE machine with SMT=8.

#include <dlfcn.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    int dlopen_flags = RTLD_NOW|RTLD_GLOBAL;
    int(*node_of_cpu)(int cpu);
    int node = 0;
    int cpu;

    if(argc != 2) {
        printf("Usage: %s <cpu index>\n", argv[0]);
        return -1;
    }

    cpu = atoi(argv[1]);
    printf("%d\n", cpu);

    void *handle = dlopen("libnuma.so.1", dlopen_flags);
    if(handle) {
        node_of_cpu = (__typeof__(node_of_cpu))dlsym(handle, "numa_node_of_cpu");
        if(node_of_cpu) {
            node = node_of_cpu(cpu);
            printf("cpu %d belongs to node %d\n", cpu, node);
        }
    }
    return 0;
}

To compile the test: gcc test.c -ldl -o test

To run: ./test 80

Other information:

$ numactl -H
available: 4 nodes (0-1,16-17)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
node 0 size: 65115 MB
node 0 free: 2955 MB
node 1 cpus: 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 1 size: 65473 MB
node 1 free: 3313 MB
node 16 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119
node 16 size: 65457 MB
node 16 free: 9310 MB
node 17 cpus: 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 17 size: 65209 MB
node 17 free: 10999 MB
node distances:
node 0 1 16 17
  0: 10 20 40 40
  1: 20 10 40 40
 16: 40 40 10 20
 17: 40 40 20 10

Userspace tool common name: libnuma

The userspace tool has the following bit modes: 64-bit

== Comment: #4 - Kamalesh Babulal <email address hidden> - 2014-08-19 12:29:15 ==
numa_node_of_cpu
|
|_numa_node_to_cpus_v2:
                                                  Allocates and updates bitmask of /sys/devices/system/node/node%d/cpumap for node 0,1 and checks if the CPU is present/isset in the node bitmap of CPU0 and CPU1. As CPU 80 is present in node 16, it continues to allocate buffer for node 2 (basically till the configured nodes)
and tries to read /sys/devices/system/node/node2/cpumap which does not exist, so it assumes that the /sys was not mounted or some thing wrong with map file and sets all the cpus in the node2 as present.

That's the reason why your seeing the warning and wrong information about node. We should be skipping the nodes which are not set in the numa_nodes_ptr