[Regression] NUMA_BALANCING disabled on arm64

[dann frazier]

[Impact]
CONFIG_NUMA_BALANCING and CONFIG_NUMA_BALANCING_DEFAULT_ENABLED were both set to =y in hwe-x/hwe-y. This changed to =n in hwe-z, unintentionally as far as I can tell. This can lead to performance degradation on NUMA-based arm64 systems when threads migrate, and their memory accesses now suffer additional latency.

[Test Case]
At a functional level:

$ test -f /proc/sys/kernel/numabalancing

Performance:

$ perf bench numa -a
I didn't see any significant changes in the RAM-bw tests (expected).
For the convergence tests, I observed the following results, which appear to be all within reasonable variance.

Test | Balancing=n | Balancing=y
-------------------------------------
1x3 | No-Converge | No-Converge
1x4 | No-Converge | 0.576s
1x6 | No-Converge | No-Converge
2x3 | No-Converge | No-Converge
3x3 | No-Converge | No-Converge
4x4 | No-Converge | No-Converge
4x4-NOTHP| No-Converge | No-Converge
4x6 | No-Converge | No-Converge
4x8 | No-Converge | No-Converge
8x4 | No-Converge | No-Converge
8x4-NOTHP| No-Converge | No-Converge
3x1 | 0.848s | 1.212s
4x1 | 0.832s | 0.712s
8x1 | 0.792s | 0.649s
16x1 | 1.511s | 1.485s
32x1 | 0.750s | 0.899s

Finally, for the bw tests, I see significant improvements across the board:
Test | BW Improvement
-------------------------
======= Process =========
2x1 | 2.2%
3x1 | 61.4%
4x1 | 25.0%
8x1 | 104.6%
8x1-NOTHP | 107.6%
16x1 | 200.9%
======= Thread ==========
4x1 | 10.9%
8x1 | 107.4%
16x1 | 230.7%
32x1 | 239.7%
2x3 | 13.5%
4x4 | 69.2%
4x6 | 84.4%
4x8 | 79.7%
4x8-NOTHP | 152.5%
3x3 | 96.1%
5x5 | 150.2%
2x16 | 122.6%
1x32 | 40.5%

[Regression Risk]
This is changing a config only on arm64, so the regression risk will be limited to those platforms. The code we will be enabling on arm64 is already enabled on other architectures (!s390x), so has been tested within Ubuntu zesty already. This was previous also enabled on arm64 in hwe-x/hwe-y, so we can gain some confidence from that.

There is certainly a possibility that this negatively impacts performance for certain workloads on NUMA/arm64 systems. If that occurs, there is a sysctl that can be used to disable this feature.

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-zesty' to 'verification-done-zesty'. If the problem still exists, change the tag 'verification-needed-zesty' to 'verification-failed-zesty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[dann frazier (dannf)]

ubuntu@d05-3:~$ cat /proc/version
Linux version 4.10.0-23-generic (buildd@bos01-arm64-039) (gcc version 6.3.0 20170406 (Ubuntu/Linaro 6.3.0-12ubuntu2) ) #25-Ubuntu SMP Fri Jun 9 09:36:27 UTC 2017
ubuntu@d05-3:~$ test -f /proc/sys/kernel/numa_balancing
ubuntu@d05-3:~$ echo $?
0

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.10.0-26.30

---------------
linux (4.10.0-26.30) zesty; urgency=low

  * linux: 4.10.0-26.30 -proposed tracker (LP: #1700528)

  * CVE-2017-1000364
    - Revert "UBUNTU: SAUCE: mm: Only expand stack if guard area is hit"
    - Revert "mm: do not collapse stack gap into THP"
    - Revert "mm: enlarge stack guard gap"
    - mm: larger stack guard gap, between vmas
    - mm: fix new crash in unmapped_area_topdown()
    - Allow stack to grow up to address space limit

linux (4.10.0-25.29) zesty; urgency=low

  * linux: 4.10.0-25.29 -proposed tracker (LP: #1699028)

  * CVE-2017-1000364
    - SAUCE: mm: Only expand stack if guard area is hit

  * CVE-2017-9074
    - ipv6: Prevent overrun when parsing v6 header options
    - ipv6: Check ip6_find_1stfragopt() return value properly.

  * [Zesty] QDF2400 ARM64 server - NMI watchdog: BUG: soft lockup - CPU#8 stuck
    for 22s! (LP: #1680549)
    - iommu/dma: Stop getting dma_32bit_pfn wrong
    - iommu/dma: Implement PCI allocation optimisation
    - iommu/dma: Convert to address-based allocation
    - iommu/dma: Clean up MSI IOVA allocation
    - iommu/dma: Plumb in the per-CPU IOVA caches
    - iommu/iova: Fix underflow bug in __alloc_and_insert_iova_range

  * Zesty update to 4.10.17 stable release (LP: #1692898)
    - xen: adjust early dom0 p2m handling to xen hypervisor behavior
    - target: Fix compare_and_write_callback handling for non GOOD status
    - target/fileio: Fix zero-length READ and WRITE handling
    - iscsi-target: Set session_fall_back_to_erl0 when forcing reinstatement
    - usb: xhci: bInterval quirk for TI TUSB73x0
    - usb: host: xhci: print correct command ring address
    - USB: serial: ftdi_sio: add device ID for Microsemi/Arrow SF2PLUS Dev Kit
    - USB: Proper handling of Race Condition when two USB class drivers try to
      call init_usb_class simultaneously
    - USB: Revert "cdc-wdm: fix "out-of-sync" due to missing notifications"
    - staging: vt6656: use off stack for in buffer USB transfers.
    - staging: vt6656: use off stack for out buffer USB transfers.
    - staging: gdm724x: gdm_mux: fix use-after-free on module unload
    - staging: wilc1000: Fix problem with wrong vif index
    - staging: comedi: jr3_pci: fix possible null pointer dereference
    - staging: comedi: jr3_pci: cope with jiffies wraparound
    - usb: misc: add missing continue in switch
    - usb: gadget: legacy gadgets are optional
    - usb: Make sure usb/phy/of gets built-in
    - usb: hub: Fix error loop seen after hub communication errors
    - usb: hub: Do not attempt to autosuspend disconnected devices
    - x86/boot: Fix BSS corruption/overwrite bug in early x86 kernel startup
    - selftests/x86/ldt_gdt_32: Work around a glibc sigaction() bug
    - x86, pmem: Fix cache flushing for iovec write < 8 bytes
    - um: Fix PTRACE_POKEUSER on x86_64
    - perf/x86: Fix Broadwell-EP DRAM RAPL events
    - KVM: x86: fix user triggerable warning in kvm_apic_accept_events()
    - KVM: arm/arm64: fix races in kvm_psci_vcpu_on
    - arm64: KVM: Fix decoding of Rt/Rt2 when trapping AArch32 CP accesses
    - block: fix blk_integrity_register to use templ...