Comment 11 for bug 1846467

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/799239
Committed: https://opendev.org/openstack/kolla-ansible/commit/95de850250ab5394a7d1ac4fdf2920fda3dea77c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 95de850250ab5394a7d1ac4fdf2920fda3dea77c
Author: John Garbutt <email address hidden>
Date: Mon Apr 27 10:59:06 2020 +0100

    Reduce RabbitMQ busy waiting, lowering CPU load

    On machines with many cores, we were seeing excessive CPU load on systems
    that were not very busy. With the following Erlang VM argument we saw
    RabbitMQ CPU usage drop from about 150% to around 20%, on a system with
    40 hyperthreads.

        +S 2:2

    By default RabbitMQ starts N schedulers where N is the number of CPU
    cores, including hyper-threaded cores. This is fine when you assume all
    your CPUs are dedicated to RabbitMQ. Its not a good idea in a typical
    Kolla Ansible setup. Here we go for two scheduler threads.
    More details can be found here:
    https://www.rabbitmq.com/runtime.html#scheduling
    and here:
    https://erlang.org/doc/man/erl.html#emulator-flags

        +sbwt none

    This stops busy waiting of the scheduler, for more details see:
    https://www.rabbitmq.com/runtime.html#busy-waiting
    Newer versions of rabbit may need additional flags:
    "+sbwt none +sbwtdcpu none +sbwtdio none"
    But this patch should be back portable to older versions of RabbitMQ
    used in Train and Stein.

    Note that information on this tuning was found by looking at data from:
    rabbitmq-diagnostics runtime_thread_stats
    More details on that can be found here:
    https://www.rabbitmq.com/runtime.html#thread-stats

    Related-Bug: #1846467

    Change-Id: Iced014acee7e590c10848e73feca166f48b622dc
    (cherry picked from commit 70f6f8e4c02ed6a8687d2bd714d3fe0b9d04d84a)