Nova feature numa-aware-vswitches not enabled

[Brent Rowsell]

Brief Description
-----------------
The following nova feature https://review.openstack.org/#/c/541290/18/specs/rocky/approved/numa-aware-vswitches.rst is not enabled in StarlingX.

This was on the planned list for StarlingX migration to Stein. Without this feature enabled there is no mechanism to ensure VM's are pinned to the same numa node as the data networks/vswitch.

Helm system overrides need to be generated for each hypervisor instance based on the system configuration. Ex:

nova.conf:
    [neutron]
    physnets = physnet0,physnet1

    [neutron_physnet_physnet0]
    numa_nodes = 0

    [neutron_physnet_physnet1]
    numa_nodes = 0,1

Severity
--------
Major

Steps to Reproduce
------------------
N/A

Expected Behavior
------------------
N/A

Actual Behavior
----------------
N/A

Reproducibility
---------------
100%

System Configuration
--------------------
Any multi-numa worker node

Branch/Pull Time/Commit
-----------------------
Current Load

Last Pass
---------
Never

Timestamp/Logs
--------------
N/A

Test Activity
-------------
N/A

[Ghada Khalil (gkhalil)]

Marking as stx.2.0. This was identified as one of the items to be supported as part of the migration to Stein. As per above, a nova helm override is required to enable this capability.

[yong hu (yhu6)]

@shuquan, since this is a follow-up of Nova upstream feature, could you pls have someone working on this?

[Zhang Kunpeng (zhangkunpeng)]

I have a question, how to assign the numa_node of those physnets? Are the nume_nodes of ports which the datanetwork bound？ Please help to check if the following two cases are right or not. Thanks.

Case 1,
system host-port-list compute-0
  ---------------------------------------
  | port | processor |
  ---------------------------------------
  | enp175s0f0 | 1 |
  | enp175s0f1 | 1 |
  | enp59s0f0 | 0 |
  | enp59s0f1 | 0 |
  ---------------------------------------
system interface-datanetwork-list compute-0
+--------------------------------------+------------+------------------+
| uuid | ifname | datanetwork_name |
+--------------------------------------+------------+------------------+
| 5b2acf23-a635-453c-899e-690eb0228e1c | enp175s0f0 | physnet1 |
| 853c3728-276f-4da9-ac83-2436e8955a3b | enp59s0f0 | physnet0 |
+--------------------------------------+------------+------------------+

nova.conf will be set
    [neutron]
    physnets = physnet0,physnet1

    [neutron_physnet_physnet0]
    numa_nodes = 0

    [neutron_physnet_physnet1]
    numa_nodes = 1

Case 2
system host-port-list compute-0
  ---------------------------------------
  | port | processor |
  ---------------------------------------
  | enp175s0f0 | 1 |
  | enp175s0f1 | 1 |
  | enp59s0f0 | 0 |
  | enp59s0f1 | 0 |
  ---------------------------------------
system interface-datanetwork-list compute-0
+--------------------------------------+------------+------------------+
| uuid | ifname | datanetwork_name |
+--------------------------------------+------------+------------------+
| 5b2acf23-a635-453c-899e-690eb0228e1c | enp175s0f0 | physnet0 |
| 853c3728-276f-4da9-ac83-2436e8955a3b | enp59s0f0 | physnet0 |
+--------------------------------------+------------+------------------+

nova.conf will be set:
    [neutron]
    physnets = physnet0

    [neutron_physnet_physnet0]
    numa_nodes = 0,1

[Brent Rowsell (brent-rowsell)]

Yes, this is correct.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/672929

[Zhang Kunpeng (zhangkunpeng)]

@Brent Rowsell
Do u know how to test this feature? I had generated these configurations in nova.conf as below,

    [neutron]
    physnets = physnet0,physnet1

    [neutron_physnet_physnet0]
    numa_nodes = 0

    [neutron_physnet_physnet1]
    numa_nodes = 1

I create 2 VMs, one with physnets0 and another with physnets1. However, these were all scheduled in the same numa node. I don't know why, could you help me for testing?

[Brent Rowsell (brent-rowsell)]

Just to clarify, you created a VM with tenant network(s) only on physnet0 and the VM got scheduled on numa1 ?

[Zhang Kunpeng (zhangkunpeng)]

yes, it seems like the feature is not enabled just changing these configurations.

[Brent Rowsell (brent-rowsell)]

According to the nova docs this is all that is required.
https://docs.openstack.org/nova/stein/admin/networking.html

Would suggest following up with the nova team.

[Zhang Kunpeng (zhangkunpeng)]

When studying the nova docs, https://docs.openstack.org/nova/stein/admin/networking.html, I found the below descriptions.

Consider an L2-type network using the Linux Bridge mechanism driver. As noted in the neutron documentation, physets are mapped to interfaces using the [linux_bridge] physical_interface_mappings configuration option. For example:
[linux_bridge]
physical_interface_mappings = provider:PROVIDER_INTERFACE

Once you have the device name, you can query sysfs to retrieve the NUMA affinity for this device. For example:
$ cat /sys/class/net/PROVIDER_INTERFACE/device/numa_node

However, there are not physical_interface_mappings configuration in stx, only '[ovs] bridge_mappings = physnet0:br-phy0,physnet1:br-phy1,' is set. And numa_node files are not found.

controller-0:~# cat /sys/class/net/br-phy0/device/numa_node
cat: /sys/class/net/br-phy0/device/numa_node: No such file or directory
controller-0:~# cat /sys/class/net/br-phy1/device/numa_node
cat: /sys/class/net/br-phy1/device/numa_node: No such file or directory

The physical interfaces' numa_node files are existed.
enp59s0f0 <-> br-phy0
enp157s0f1 <-> br-phy1

controller-0:~# cat /sys/class/net/enp59s0f0/device/numa_node
0
controller-0:~# cat /sys/class/net/enp175s0f1/device/numa_node
1

So are the configurations or someting correct？

[Brent Rowsell (brent-rowsell)]

Hi Kunpeng,

According to the spec

Physical networks
  Physical networks, or *physnet*\s, are identified through the use of an
  arbitrary label. In the case of the Open vSwitch (OVS) ML2 driver, these
  labels are mapped to a given OVS bridge containing the physical interfaces
  using the ``[ovs] bridge_mappings`` neutron configuration option. For
  example:

  .. code-block:: ini
     :caption: openvswitch_agent.ini

     [ovs]
     bridge_mappings = provider:br-provider

  This will map the *physnet* ``provider`` to an OVS bridge ``br-provider``. It
  is expected that this bridge will contain a logical interface (you can use
  bonded NICs to provide failover).

[yong hu (yhu6)]

@kunpeng, pls write down the details of your test/deployment with the bare metal server.
AIO Simplex
NICs:
VM tenant with NIC (??) on which numa node

In addition, you can study the feature spec: https://review.openstack.org/#/c/541290
and understand what the expected behavior should be, to figure out:
1). possibly there is a gap/issue in the original implementation.
2). what you can do to improve your own patch.

[Zhang Kunpeng (zhangkunpeng)]

I achieved in starlingx, the feature worked well with this patch.
The reason for the previous failure is that the flavors are not set with property "hw:numa_nodes".

The property is needed, however, it doesn't say in the guide[1] or spec.

Then I think this patch[2] can be marged.:)

[1]https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/numa-aware-vswitches.html
[2]https://review.opendev.org/#/c/672929/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672929
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=1a923c8474b0e4d7ef78b3444d131682babfe6aa
Submitter: Zuul
Branch: master

commit 1a923c8474b0e4d7ef78b3444d131682babfe6aa
Author: zhangkunpeng <[e-mail]>
Date: Fri Jul 26 16:34:44 2019 +0800

    Generate configuration option to enable numa-aware-vswitches

    Generate physnets configuration option in neutron group and generate
    neutron physnet configuration groups that specify the mapping of each
     physnet to colocated numa nodes on the host.

    nova.conf for example:
        [neutron]
        physnets = physnet0,physnet1

        [neutron_physnet_physnet0]
        numa_nodes = 0

        [neutron_physnet_physnet1]
        numa_nodes = 0,1

        [neutron_tunneled]
        numa_nodes = 0

    Change-Id: Iaa633fb19b2422cf6f0331f43c7b833200a5c159
    Closes-Bug: 1837232
    Signed-off-by: zhangkunpeng <[e-mail]>

[Ghada Khalil (gkhalil)]

@Kungpeng, please cherrypick to the r/stx.2.0 branch before 2019-08-23

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/677325

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/677325
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=c0ce04ba163649d6e8bad224075d8412af81f163
Submitter: Zuul
Branch: r/stx.2.0

commit c0ce04ba163649d6e8bad224075d8412af81f163
Author: zhangkunpeng <[e-mail]>
Date: Fri Jul 26 16:34:44 2019 +0800

    Generate configuration option to enable numa-aware-vswitches

    Generate physnets configuration option in neutron group and generate
    neutron physnet configuration groups that specify the mapping of each
     physnet to colocated numa nodes on the host.

    nova.conf for example:
        [neutron]
        physnets = physnet0,physnet1

        [neutron_physnet_physnet0]
        numa_nodes = 0

        [neutron_physnet_physnet1]
        numa_nodes = 0,1

        [neutron_tunneled]
        numa_nodes = 0

    Change-Id: Iaa633fb19b2422cf6f0331f43c7b833200a5c159
    Closes-Bug: 1837232
    Signed-off-by: zhangkunpeng <[e-mail]>
    (cherry picked from commit 1a923c8474b0e4d7ef78b3444d131682babfe6aa)