Fix calculation of erlang thread pool size for rabbitmq

Bug #1573696 reported by Alexey Lebedeff on 2016-04-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Alexey Lebedeff
7.0.x
Medium
MOS Maintenance
8.0.x
Medium
MOS Maintenance
Mitaka
Medium
Alexey Lebedeff

Bug Description

Detailed bug description:
  https://bugs.launchpad.net/fuel/+bug/1472793 introduced automated guessing of erlang thread pool size for rabbit. But rabbitmq upstream is currently using another algorithm for doing this calculation - https://github.com/rabbitmq/rabbitmq-server/issues/151

  rabbit uses separate param RABBITMQ_IO_THREAD_POOL_SIZE to control this variable, but fuel controls the same value through RABBITMQ_SERVER_ERL_ARGS. As a result erlang runtime receives two contradictory values for pool size, but I'm not sure which of them is actually used.

  We need:
  - Switch fuel to using RABBITMQ_SERVER_ERL_ARGS (it'll prevent duplicate args and override rabbitmq default calculated thread pool size)
  - Maybe update defaults calculation in fuel, so they'll be the same

Steps to reproduce:

Run following command on a controller and observe that erlang runtime was passed 2 '-A' options:

ps ax | grep -- '-sname rabbit' | perl -nE 'for (/-A\s*\d+/g) { say } '
-A 64
-A18

Expected results:
Only one '-A' option for beam.smp, with value calculated according to chosen algorithm

Impact:
  Machine with big number of CPU cores may be underutilized by rabbit

Dmitry Klenov (dklenov) on 2016-04-25
Changed in fuel:
assignee: nobody → MOS Oslo (mos-oslo)
tags: added: area-mos
Changed in fuel:
milestone: none → 10.0
importance: Undecided → Medium
status: New → Confirmed
Changed in fuel:
assignee: MOS Oslo (mos-oslo) → Alexey Lebedeff (alebedev-a)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/332234
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0c16cbc9a723edfc7a9edd08faac8f5fc1b974bf
Submitter: Jenkins
Branch: master

commit 0c16cbc9a723edfc7a9edd08faac8f5fc1b974bf
Author: Alexey Lebedeff <email address hidden>
Date: Tue Jun 21 18:58:51 2016 +0300

    Remove erlang/rabbit thread pool calculation

    Proper version of this calculation is implemented upstream -
    https://github.com/rabbitmq/rabbitmq-server/issues/151

    And removed version of the code is actually harmful, as it's using
    physicalprocessorcount (i.e. number of CPU sockets) for
    calculations. So on a 2 CPU/48 thread system there it was 30 threads
    instead of 768 calculated upstream.

    I've decided that it's easier to completely remove this code instead
    of duplicating current formula in upstream and providing a way to
    override this value through hiera - just not worth a hassle.

    Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
    Closes-Bug: 1573696

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/403740
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=63d73db1164da00d032759734a2acb49d95e984e
Submitter: Jenkins
Branch: stable/mitaka

commit 63d73db1164da00d032759734a2acb49d95e984e
Author: Alexey Lebedeff <email address hidden>
Date: Tue Jun 21 18:58:51 2016 +0300

    Remove erlang/rabbit thread pool calculation

    Proper version of this calculation is implemented upstream -
    https://github.com/rabbitmq/rabbitmq-server/issues/151

    And removed version of the code is actually harmful, as it's using
    physicalprocessorcount (i.e. number of CPU sockets) for
    calculations. So on a 2 CPU/48 thread system there it was 30 threads
    instead of 768 calculated upstream.

    I've decided that it's easier to completely remove this code instead
    of duplicating current formula in upstream and providing a way to
    override this value through hiera - just not worth a hassle.

    Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
    Closes-Bug: 1573696
    (cherry picked from commit 0c16cbc9a723edfc7a9edd08faac8f5fc1b974bf)

tags: added: in-stable-mitaka

Reviewed: https://review.openstack.org/396677
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7fa632dfdfc53a1381cde2b7af634cee6479483d
Submitter: Jenkins
Branch: stable/8.0

commit 7fa632dfdfc53a1381cde2b7af634cee6479483d
Author: Alexey Lebedeff <email address hidden>
Date: Fri Nov 11 18:41:34 2016 +0300

    Remove erlang/rabbit thread pool calculation

    Proper version of this calculation is implemented upstream -
    https://github.com/rabbitmq/rabbitmq-server/issues/151

    And removed version of the code is actually harmful, as it's using
    physicalprocessorcount (i.e. number of CPU sockets) for
    calculations. So on a 2 CPU/48 thread system there it was 30 threads
    instead of 768 calculated upstream.

    I've decided that it's easier to completely remove this code instead
    of duplicating current formula in upstream and providing a way to
    override this value through hiera - just not worth a hassle.

    Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
    Closes-Bug: 1573696

no longer affects: fuel/newton
tags: added: on-verification
TatyanaGladysheva (tgladysheva) wrote :

Verified on 8.0 + MU4 updates.

Before the fix:
root@node-1:~# ps ax | grep -- '-sname rabbit' | perl -nE 'for (/-A\s*\d+/g) { say } '
-A 64
-A4

After the fix:
root@node-1:~# ps ax | grep -- '-sname rabbit' | perl -nE 'for (/-A\s*\d+/g) { say } '
-A 64

tags: removed: on-verification

Reviewed: https://review.openstack.org/396686
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=2858871ff02b0637e5df9e17ce26f8217cf38071
Submitter: Jenkins
Branch: stable/7.0

commit 2858871ff02b0637e5df9e17ce26f8217cf38071
Author: Alexey Lebedeff <email address hidden>
Date: Fri Nov 11 19:20:23 2016 +0300

    Remove erlang/rabbit thread pool calculation

    Proper version of this calculation is implemented upstream -
    https://github.com/rabbitmq/rabbitmq-server/issues/151

    And removed version of the code is actually harmful, as it's using
    physicalprocessorcount (i.e. number of CPU sockets) for
    calculations. So on a 2 CPU/48 thread system there it was 30 threads
    instead of 768 calculated upstream.

    I've decided that it's easier to completely remove this code instead
    of duplicating current formula in upstream and providing a way to
    override this value through hiera - just not worth a hassle.

    Change-Id: I415d446af0a822d2a5ce3478fd9db1dd0f13e115
    Closes-Bug: 1573696

TatyanaGladysheva (tgladysheva) wrote :

Verified on 7.0 + MU8 updates.

After the fix:
root@node-4:~# ps ax | grep -- '-sname rabbit' | perl -nE 'for (/-A\s*\d+/g) { say } '
-A 64

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.