AIO: Too many rabbit threads

[Brent Rowsell]

On AIO, an erlang scheduler thread is launched per logical core. While this probably makes sense on a standard controller, it does not make sense on an AIO. On AIO there should be a scheduler thread per platform core.

Also the number of aync threads needs to be reviewed as well. There are 64 on a 56 logical core machine with AIO configured.

[yong hu (yhu6)]

checked with a Simplex deployment on bare metal sever, there were 8 cores (/proc/cpuinfo) and 77 threads under beam.smp (Erlang scheduler) process.

Similarly with a multi-node deployment, on virtual server, on controller-0, there were only 6 cores but even more threads (123):
---------------------------------------------------------
controller-0:~$ cat /proc/cpuinfo | grep "processor"
processor : 0
processor : 1
processor : 2
processor : 3
processor : 4
processor : 5

controller-0:~$ ps aux | grep rabbitmq | grep -v grep
root 559463 0.0 0.0 4620 736 ? Ss 6月27 0:01 /bin/sh /opt/rabbitmq/sbin/rabbitmq-server
root 574366 15.3 0.8 4629392 272896 ? Sl 6月27 1241:13 /usr/local/lib/erlang/erts-10.3.1/bin/beam.smp -W w -A 96 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -B i -- -root /usr/local/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /opt/rabbitmq/ebin -noshell -noinput -s rabbit boot -name <email address hidden> -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /opt/rabbitmq/sbin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger tty -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file tty -rabbit lager_upgrade_file tty -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/opt/rabbitmq/plugins" -rabbit plugins_expand_dir "/<email address hidden>" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/<email address hidden>" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

controller-0:~$ ps huH p 574366 | wc -l
123

--------------------------------------------------------

Need to do research how to adjust the number of threads for beam.smp.

@Bin, please look into this topic.

[Ghada Khalil (gkhalil)]

Marking as stx.2.0 gating -- AIO configuration/performance optimization

[Bin Yang (byangintel)]

I reproduced it on my machine (6 CPUs)

From "systemctl status":
=======================
/usr/lib64/erlang/erts-7.3.1.2/bin/beam.smp -W w -A 96 ...
It Sets the number of threads in async thread pool as "96"

From /usr/lib/rabbitmq/bin/rabbitmq-server:
===========================================
    if [ -z "${RABBITMQ_IO_THREAD_POOL_SIZE}" ]; then
        RABBITMQ_IO_THREAD_POOL_SIZE=$(
            ${ERL_DIR}erl -pa "$RABBITMQ_EBIN_ROOT" \
                      -boot "${CLEAN_BOOT_FILE}" \
                      -noinput \
                      -s rabbit_misc report_default_thread_pool_size
        )
    fi

From rabbit_common/src/rabbit_misc.erl
======================================
%% Discussion of choosen values is at
%% https://github.com/rabbitmq/rabbitmq-server/issues/151
guess_default_thread_pool_size() ->
    PoolSize = 16 * guess_number_of_cpu_cores(),
    min(1024, max(64, PoolSize)).

report_default_thread_pool_size() ->
    io:format("~b", [guess_default_thread_pool_size()]),
    erlang:halt(0),
    ok.

Conclusion:
===========
The thread pool size (16 * 6 = 96) is by design.
All these threads are idle if no workload.

I think it should not be a issue.

[Bin Yang (byangintel)]

Do we want to calculate the CPU number based on "PLATFORM_CPU_LIST" on AIO?

[Brent Rowsell (brent-rowsell)]

Yes, we want to base the number of threads based on the platform cores.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671177

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671651

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669562
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=4c5ff8f62790c01a273f088bfef5b2b6e840ecba
Submitter: Zuul
Branch: master

commit 4c5ff8f62790c01a273f088bfef5b2b6e840ecba
Author: Bin Yang <email address hidden>
Date: Sun Jul 7 10:39:40 2019 +0800

    limit rabbitmq CPU usage

    On AIO, the rabbitmq CPU usage should be based on the platform cores.

    Change-Id: Ic73a5c82cd37cc0c8e3119577842a10fb1167a69
    Closes-Bug: 1834796
    Signed-off-by: Bin Yang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671651
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=a30e09bb87186c92953b3d67c0bf568c3707390e
Submitter: Zuul
Branch: master

commit a30e09bb87186c92953b3d67c0bf568c3707390e
Author: Bin Yang <email address hidden>
Date: Fri Jul 19 11:17:55 2019 +0800

    add get_platform_cpus

    Add helper function for platform cpu number calculation

    Change-Id: I6f656594e80fb067794cc14ba8f01db84585d198
    Partial-Bug: 1834796
    Signed-off-by: Bin Yang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671177
Committed: https://git.openstack.org/cgit/starlingx/upstream/commit/?id=3209f7cdad1df7b9e36990c43a7bee9101d77b6d
Submitter: Zuul
Branch: master

commit 3209f7cdad1df7b9e36990c43a7bee9101d77b6d
Author: Bin Yang <email address hidden>
Date: Wed Jul 17 11:09:01 2019 +0800

    calculate rabbitmq io thread pool size based on platform cores

    Change-Id: I2d73c52a2e17b4519350229aa95d6766bd4b7129
    Depends-On: https://review.opendev.org/#/c/671651
    Closes-Bug: 1834796
    Signed-off-by: Bin Yang <email address hidden>

[Al Bailey (albailey1974)]

Tested this change in an AIO-DX env.

The helm overrides are set to:

      resources:
        enabled: true
        prometheus_rabbitmq_exporter:
          limits:
            cpu: 2000m
            memory: 16500Mi
        server:
          limits:
            cpu: 2000m
            memory: 16500Mi

However the beam process is still using 600 threads.

ps -e -o pid,nlwp,comm,command --sort=-nlwp|cut -c1-160|head -n 1
    PID NLWP COMMAND COMMAND
 641323 631 beam.smp /usr/local/lib/erlang/erts-10.3.1/bin/beam.smp -W w -A 544 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 104

[Brent Rowsell (brent-rowsell)]

Re-opening as the change did not address the issue.

[Bin Yang (byangintel)]

It looks like your test does not consider cgroup quota.

The cpu quota can be find with below steps:
1. cat /proc/<pid>/cgroup | grep cpu #get cgroup namespace
2. cfs_period_us=`cat /sys/fs/cgroup/cpu/<cgroup>/cpu.cfs_period_us`
   cfs_quota_us=`cat /sys/fs/cgroup/cpu/<cgroup>/cpu.cfs_quota_us`
3. the real cpu usage is 'cfs_quota_us/cfs_period_us'

You can run heavy workload to very it.
I think the CPU% of rabbitmq should not > (cfs_quota_us/cfs_period_us)*100%

[Brent Rowsell (brent-rowsell)]

Your change bound the cpu usage of the pod to 2 cores. There are only 2 cores.
This LP asked to shrink the footprint of rabbit based on the number of cores it has available (i.e. fewer threads).

[Bin Yang (byangintel)]

It always uses cgroup for CPU quota in K8S. And openstack helm charts always use it for CPU/Mem resources limits.
Here is the general snippet which is used for most of openstack helm charts (including rabbitmq):
https://github.com/openstack/openstack-helm-infra/blob/master/helm-toolkit/templates/snippets/_kubernetes_resources.tpl

It’s meaningless to just talk about "thread number" without considering "cgroup CPU quota". So my suggestion is to update the test cases to calculate real CPU usage considering the quota.

[Brent Rowsell (brent-rowsell)]

We need to reduce the CPU usage of rabbit in the AIO configs, your change does nothing to address this. Changing the testcase is not the answer.

[Bin Yang (byangintel)]

why you think this patch does nothing? with this patch, the cpu% of rabbitmq will be limited to (100*plt_cpu_num)%.

And it is the design of K8S and openstack helm charts infra. Do we need a new design for CPU usage quota?

[Bin Yang (byangintel)]

Please refer to the k8s document: https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit

Without this patch, the max cpu usage of rabbitmq will be (100*total_cpu_num)%.
After applying this patch, the max cpu usage of rabbitmq will be (100*plt_cpu_num)%.

"CPU limit" is the design of K8S for CPU usage quota and rabbitmq is one of K8s services.

[Frank Miller (sensfan22)]

Your original patch is helpful but just not sufficient to address the AIO performance issues currently seen. The AIO configuration is cpu bound and is currently maxed out on the 2 platform cores used by AIO. As part of investigation for https://bugs.launchpad.net/starlingx/+bug/1837426, one of the observations is that rabbit is using 631 threads for its beam process. This is too many as the AIO is limited to 2 cores for all platform processes. An additional change will be needed to reduce the number of beam threads when using only 2 cores. Recommendation is to re-assign this LP to the developers involved in 1837426 as they are identifying multiple options to reduce overall platform cpu usage.

[Bin Yang (byangintel)]

In LP#1837426: "the platform CPU usage on the active controller is very high (90-100%) resulting in major/critical CPU alarms."

CPU% is NOT thread number !!!

cgroup CPU quota is design to limit the CPU%. And K8S uses it for CPU resource limit. please refer to the document https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit

It is easy to create patch to change the thread number. But what issue do we really want to fix? thread number or cpu usage?

[Cindy Xie (xxie1)]

@Bin, I think what Frank and Al who working on 1837426 are trying to optimize CPU% and think reducing # of rabbitMQ threads is one of the way to address CPU%. If there is a way to make the patch to reduce rabbitMQ thread#, can you please make it so that Al can test from his side?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676031
Committed: https://git.openstack.org/cgit/starlingx/upstream/commit/?id=263741c9769ac27374aa10033c0bbb57b9511348
Submitter: Zuul
Branch: master

commit 263741c9769ac27374aa10033c0bbb57b9511348
Author: Bin Yang <email address hidden>
Date: Tue Aug 13 09:09:51 2019 +0800

    add io_thread_pool for rabbitmq

    Here is the upstream patch link:
    https://review.opendev.org/#/c/677425

    Change-Id: I71b6d37b5e335ce9045937009fa93d47a49bcd4c
    Partial-Bug: 1834796
    Signed-off-by: Bin Yang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676035
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=b1b7878bcbcb0eaf7b84429ebfabe2527251f5ff
Submitter: Zuul
Branch: master

commit b1b7878bcbcb0eaf7b84429ebfabe2527251f5ff
Author: Bin Yang <email address hidden>
Date: Tue Aug 13 09:17:45 2019 +0800

    rabbitmq: change io_thread_pool based on platform cpus

    By default, rabbitmq calculates the io thread pool size based on cpus.
    On AIO, not all cpus are reserved for platform services. The pool size
    should be calculated based on platform cpus.

    Change-Id: I856e8dda6827a4c36944f43eccdc9a91b19655eb
    Depends-On: https://review.opendev.org/#/c/676031
    Closes-Bug: 1834796
    Signed-off-by: Bin Yang <email address hidden>

[Ghada Khalil (gkhalil)]

@Bin, please cherrypick the changes to the r/stx.2.0 branch before 2019-08-23

[Ghada Khalil (gkhalil)]

Please monitor the sanity results on master with the 2019-08-21 load to make sure there are no issues before proceeding with the cherry-pick.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/677591

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/677592

[Bin Yang (byangintel)]

https://review.opendev.org/#/c/677592/
https://review.opendev.org/#/c/677591/

wait for sanity result on master ...

[Ghada Khalil (gkhalil)]

Sanity on master was green using the 2019-08-21 load:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190821T053000Z/outputs/iso/

Please proceed with the merge

[Bin Yang (byangintel)]

I removed my "workflow -1" and wait for core reviewer to add "workflow +1"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/677591
Committed: https://git.openstack.org/cgit/starlingx/upstream/commit/?id=5942005fde88f7b65fc9a1c03b4f2f3d7847fcf0
Submitter: Zuul
Branch: r/stx.2.0

commit 5942005fde88f7b65fc9a1c03b4f2f3d7847fcf0
Author: Bin Yang <email address hidden>
Date: Tue Aug 13 09:09:51 2019 +0800

    add io_thread_pool for rabbitmq

    Here is the upstream patch link:
    https://review.opendev.org/#/c/677425

    Change-Id: I71b6d37b5e335ce9045937009fa93d47a49bcd4c
    Partial-Bug: 1834796
    Signed-off-by: Bin Yang <email address hidden>
    (cherry picked from commit 263741c9769ac27374aa10033c0bbb57b9511348)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/677592
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=7e6e52464e6a432f41e70c33faf2de1025a351d6
Submitter: Zuul
Branch: r/stx.2.0

commit 7e6e52464e6a432f41e70c33faf2de1025a351d6
Author: Bin Yang <email address hidden>
Date: Tue Aug 13 09:17:45 2019 +0800

    rabbitmq: change io_thread_pool based on platform cpus

    By default, rabbitmq calculates the io thread pool size based on cpus.
    On AIO, not all cpus are reserved for platform services. The pool size
    should be calculated based on platform cpus.

    Change-Id: I856e8dda6827a4c36944f43eccdc9a91b19655eb
    Depends-On: https://review.opendev.org/#/c/677591/
    Closes-Bug: 1834796
    Signed-off-by: Bin Yang <email address hidden>
    (cherry picked from commit b1b7878bcbcb0eaf7b84429ebfabe2527251f5ff)

[krishna pavan (ksamudrx)]

Verified test with simplex configuration and it got passed.

Build info:

OS="centos"
SW_VERSION="19.08"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="r/stx.2.0"

JOB="STX_BUILD_2.0"
<email address hidden>"
BUILD_NUMBER="40"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-08-26 23:30:00 +0000"