[dpdk]OpenvSwitch pmd threads fail to start due to incorrect cpu pinning if host has more than 1 NUMA node
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Committed
|
High
|
Arthur Svechnikov | ||
Mitaka |
Fix Released
|
High
|
Arthur Svechnikov | ||
Newton |
Fix Committed
|
High
|
Arthur Svechnikov |
Bug Description
Environment: MOS 9.0 ISO 370, hardware lab
Steps to reproduce:
1. Create environment, add a host with >1 NUMA nodes as compute
2. Configure hugepages and DPDK CPU pinning (min 4096 2M hugepages, at least one CPU pinned for DPDK)
3. Turn on DPDK on one interface and move Private network to this interface, deploy cluster
4. Check OVS process CPU utilization and threads on compute, e.g. with "top -n 1 -bH -p `pgrep ovs-vswitchd`"
Expected result:
There are OVS threads with name pmdXX, which fully utilize 1 CPU core each
Actual result:
No pmd threads, error message in /var/log/
2016-05-
Additional information:
PMD threads are successfully started, if NIC and all cores in pmd-cpu-mask are on same NUMA node
Additionally, format of pmd-cpu-mask causes a warning in /var/log/
2016-05-
Diagnostic snapshot: http://
tags: | added: area-python |
description: | updated |
summary: |
[dpdk]OpenvSwitch pmd threads fail to start due to incorrect cpu pinning - if hoast has more than 1 NUMA node + if host has more than 1 NUMA node |
Changed in fuel: | |
assignee: | Arthur Svechnikov (asvechnikov) → Fedor Zhadaev (fzhadaev) |
Changed in fuel: | |
assignee: | Fedor Zhadaev (fzhadaev) → Arthur Svechnikov (asvechnikov) |
To sum up: openvswitch- switch ( -c 0xXXX ) pmd-cpu- mask ). It should be without leading 0x
There are 2 CPU masks for OVS+DPDK in astute.yaml:
1) ovs_core_mask, which is populated to /etc/default/
2) ovs_pmd_core_mask, shich is populated to OVS database ( get Open_vSwitch . other_config:
For successful operation both parameters should mask CPU cores from NUMA node to which NIC is attached.