OVS n-handler-threads and n-handler-threads parameters are incoherent with hypervisor isolated core list

[Haresh]

Description
===========
All OpenStack and hypervisor services will run on the HostCpusList. The problem is that ovs-vswitchd is not aware of such partitioning, and is spawning many userland threads as it believes that he can run on all of the CPUs.

Steps to reproduce
==================
1) Deploy OVS. Observe the revalidator and handler threads
[heat-admin@compute-0 ~]$ ps H -o 'tid comm' <openvswitchd process id>
    TID COMMAND
  22031 ovs-vswitchd
  22032 ovs-vswitchd
  22034 ovs-vswitchd
  22035 ovs-vswitchd
  22036 ovs-vswitchd
  22037 ovs-vswitchd
  22038 dpdk_watchdog1
  22046 urcu2
  22066 ct_clean3
  22739 pmd164
  22740 pmd165
  22741 pmd167
  22742 pmd166
 493539 handler613
 493540 revalidator610 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 493541 revalidator611 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 493542 revalidator612 <<<<<<<<<<<<<<<<<<<<<<<<<<<
 493543 revalidator614 <<<<<<<<<<<<<<<<<<<<<<<<<<

Expected result
===============
Running on specified cores

Actual result
=============
Utilizing all available cores

Environment
===========
OVS with DPDK
Logs & Configs
sudo ovs-vsctl get Open_vSwitch . other_config:n-revalidator-threads
sudo ovs-vsctl get Open_vSwitch . other_config:n-handler-threads
==============

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/650627

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/650627
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3c5ad2aab98fa9a1ab26cc6c348a64806276e8b9
Submitter: Zuul
Branch: master

commit 3c5ad2aab98fa9a1ab26cc6c348a64806276e8b9
Author: hakhande <email address hidden>
Date: Mon Apr 8 13:43:05 2019 +0530

    OVS Revalidator and handler threads

    Revalidator and handler threads are not coherent with lcore
    Configure these threads accoding to confgiure lcores

    Change-Id: Idc3328658a4c5c21fd011c6c4f791e7993559f1a
    Closes-Bug: #1822571
    Depends-On: https://review.openstack.org/650626

[Jeremy Stanley (fungi)]

It's unclear why you switched this report from Public to Public Security. Neither the bug description nor the patch indicate a suspected attack vector which could take advantage of it. For example, does this represent a possible denial of service condition which an unprivileged user can exploit?

[Haresh (hakhande)]

Hi,

This was accidental change. I will revert back to original. Thanks for
bringing it to my attention.

Thanks
Haresh

On Sat 25 May, 2019, 9:05 PM Jeremy Stanley, <email address hidden> wrote:

> It's unclear why you switched this report from Public to Public
> Security. Neither the bug description nor the patch indicate a suspected
> attack vector which could take advantage of it. For example, does this
> represent a possible denial of service condition which an unprivileged
> user can exploit?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1822571
>
> Title:
> OVS n-handler-threads and n-handler-threads parameters are incoherent
> with hypervisor isolated core list
>
> Status in tripleo:
> Fix Released
>
> Bug description:
> Description
> ===========
> All OpenStack and hypervisor services will run on the HostCpusList. The
> problem is that ovs-vswitchd is not aware of such partitioning, and is
> spawning many userland threads as it believes that he can run on all of the
> CPUs.
>
> Steps to reproduce
> ==================
> 1) Deploy OVS. Observe the revalidator and handler threads
> [heat-admin@compute-0 ~]$ ps H -o 'tid comm' <openvswitchd process id>
> TID COMMAND
> 22031 ovs-vswitchd
> 22032 ovs-vswitchd
> 22034 ovs-vswitchd
> 22035 ovs-vswitchd
> 22036 ovs-vswitchd
> 22037 ovs-vswitchd
> 22038 dpdk_watchdog1
> 22046 urcu2
> 22066 ct_clean3
> 22739 pmd164
> 22740 pmd165
> 22741 pmd167
> 22742 pmd166
> 493539 handler613
> 493540 revalidator610 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> 493541 revalidator611 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> 493542 revalidator612 <<<<<<<<<<<<<<<<<<<<<<<<<<<
> 493543 revalidator614 <<<<<<<<<<<<<<<<<<<<<<<<<<
>
> Expected result
> ===============
> Running on specified cores
>
> Actual result
> =============
> Utilizing all available cores
>
> Environment
> ===========
> OVS with DPDK
> Logs & Configs
> sudo ovs-vsctl get Open_vSwitch . other_config:n-revalidator-threads
> sudo ovs-vsctl get Open_vSwitch . other_config:n-handler-threads
> ==============
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/tripleo/+bug/1822571/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/661566

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/661763

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-vswitch 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/664600

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/661566
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1797d56cc59c2db92b87b8fe2db7b3f80e4279f6
Submitter: Zuul
Branch: stable/stein

commit 1797d56cc59c2db92b87b8fe2db7b3f80e4279f6
Author: hakhande <email address hidden>
Date: Mon Apr 8 13:43:05 2019 +0530

    OVS Revalidator and handler threads

    Revalidator and handler threads are not coherent with lcore
    Configure these threads according to configure lcores

    Change-Id: Idc3328658a4c5c21fd011c6c4f791e7993559f1a
    Closes-Bug: #1822571
    Depends-On: https://review.openstack.org/661762
    (cherry picked from commit 3c5ad2aab98fa9a1ab26cc6c348a64806276e8b9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/661763
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=27de28a2c3faaa366ef23670679cc35067e57c0d
Submitter: Zuul
Branch: stable/rocky

commit 27de28a2c3faaa366ef23670679cc35067e57c0d
Author: hakhande <email address hidden>
Date: Mon Apr 8 13:43:05 2019 +0530

    OVS Revalidator and handler threads

    Revalidator and handler threads are not coherent with lcore
    Configure these threads accoding to confgiure lcores

    Change-Id: Idc3328658a4c5c21fd011c6c4f791e7993559f1a
    Closes-Bug: #1822571
    Depends-On: https://review.openstack.org/661764
    (cherry picked from commit 3c5ad2aab98fa9a1ab26cc6c348a64806276e8b9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664600
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=87ba89813397bad46b9b7ce377b4452033a36874
Submitter: Zuul
Branch: stable/queens

commit 87ba89813397bad46b9b7ce377b4452033a36874
Author: hakhande <email address hidden>
Date: Mon Apr 8 13:43:05 2019 +0530

    OVS Revalidator and handler threads

    Revalidator and handler threads are not coherent with lcore
    Configure these threads accoding to confgiure lcores

    Conflicts:
            puppet/services/openvswitch.yaml

    Change-Id: Idc3328658a4c5c21fd011c6c4f791e7993559f1a
    Closes-Bug: #1822571
    Depends-On: https://review.opendev.org/#/c/664239/
    (cherry picked from commit 3c5ad2aab98fa9a1ab26cc6c348a64806276e8b9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-vswitch queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-vswitch rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-vswitch stein-eol release.