Nvme interrupts not affined to platform cores on aio-dx

[Brent Rowsell]

Brief Description
-----------------
Disk interrupts should be affined to the platform cores. On AIO-DX I am seeing interrupts affined to the application cores. Platform cores in this case are 0 and 1.

 irq/279-nvme1q2 5686
 irq/307-nvme2q2 5699
 irq/335-nvme0q2 6178

taskset -apc 5686
pid 5686's current affinity list: 27
controller-0:~$ taskset -apc 5699
pid 5699's current affinity list: 27
controller-0:~$ taskset -apc 6178
pid 6178's current affinity list: 27

They can be manually re-affined

sudo taskset -pc 0 5686
Password:
pid 5686's current affinity list: 27
pid 5686's new affinity list: 0

Severity
--------
Major, will cause application jitter

Steps to Reproduce
------------------
Unlock node

Expected Behavior
------------------
Interrupts are affined to plaform cores

Actual Behavior
----------------
Interrupts affined to an application cores

Reproducibility
---------------
TBD

System Configuration
--------------------
AIO-DX

Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2019-09-11 14:52:32 -0400"

Last Pass
---------
Not sure

Timestamp/Logs
--------------

Test Activity
-------------
Developer Testing

[Ghada Khalil (gkhalil)]

Marking stx.3.0 / medium priority - issue impacts system performance

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/685976

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685976
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=45a149497ee8d05b571328214c2c819c63ea9ea9
Submitter: Zuul
Branch: master

commit 45a149497ee8d05b571328214c2c819c63ea9ea9
Author: Jim Gauld <email address hidden>
Date: Tue Oct 1 09:39:57 2019 -0400

    Affine NVMe disk interrupts to platform cores

    This affines NVMe disk interrupts to platform cores for low-latency
    worker configurations to reduce application jitter. Tasks with names
    matching irq/<pid>-nvme* get re-affined.

    This also corrects writing of related affinity settings for:
    - /proc/irq/default_smp_affinity
    - /sys/devices/virtual/workqueue/cpumask
    - /sys/bus/workqueue/devices/writeback/cpumask

    The platform coremasks require being delimited with commas every 8 hex
    characters. HT-enabled hosts tend to have very large affinity masks,
    e.g., host with 88 CPUs, the coremask for the cpulist 0-1,44-45 is:
    00000000,00000000,00000000,00003000,00000000,00000003

    Just writing 30000000000000000003 will produce the following error:
    echo: write error: Value too large for defined data type

    Change-Id: Ibad5beffc791b796e472ff19950cf6a3e973be86
    Closes-bug: 1843942
    Signed-off-by: Jim Gauld <email address hidden>