RabbitMQ should use appropriate async thread pool size

[Alexey Lebedeff]

Detailed bug description:
  Erlang runtime has parameter controlling the size of async thread pool. Rabbitmq by default specifies the default value of 64 for this parameter. But for hosts with large number of CPU cores it could be insufficient. Detailed discussion is at https://groups.google.com/forum/#!msg/rabbitmq-users/ijHVImbazFI/cxPtzoxKEUoJ

Steps to reproduce:
Run
  ps ax | perl -nE '/-s rabbit boot/ && /^\s*(\d+).*?-A (\d+)/ && say "$2"'
on controller, resulting number should be greater than 64 if there is more than 4 CPU cores.

Expected results:
  Number between 64 and 1024, closest to 16 * number of CPU cores.

Actual result:
  64

Reproducibility:
  Always on

Workaround:
  Specify value in rabbitmq config on a per-installation basis.

Impact:
  Severe underutilization of powerful host machines

Additional:
  There is an upstream patch about autotuning this parameter - https://github.com/rabbitmq/rabbitmq-server/issues/151
  We should backport it for 9.0

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Alexey Lebedeff <email address hidden>
Review: https://review.fuel-infra.org/19143

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Alexey Lebedeff <email address hidden>
Review: https://review.fuel-infra.org/19144

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Alexey Lebedeff <email address hidden>
Review: https://review.fuel-infra.org/19189

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Alexey Lebedeff <email address hidden>
Review: https://review.fuel-infra.org/19192

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19189
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 345c57dde761e0be9992ac3fde3c6c6a560569e6
Author: Alexey Lebedeff <email address hidden>
Date: Tue Apr 5 12:12:29 2016

Autotune erlang thread pool size

Backport from upstream https://github.com/binarin/rabbitmq-server/tree/rabbitmq-server-151

Other patches are moved around and added to streamline centos and ubuntu packaging.

Change-Id: I669523ee1db5e1d30af247dd94e36cd31613c4a9
Closes-Bug: 1565829

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19144
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: 5a55ada3c36a52128039f54736664e1eac47c5e9
Author: Alexey Lebedeff <email address hidden>
Date: Mon Apr 4 17:30:31 2016

Autotune erlang thread pool size

Backport from upstream https://github.com/binarin/rabbitmq-server/tree/rabbitmq-server-151

Change-Id: I4a7cbe6e3989451bd1a306cdad9887769e035e06
Closes-Bug: 1565829

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexey Lebedeff <email address hidden> on branch: master
Review: https://review.fuel-infra.org/19143
Reason: It's already in the master branch

[Alexey Galkin (agalkin)]

Verified as fixed in 9.0-229.

Master-node.

[root@fuel ~]# nproc
6
[root@fuel ~]# ps ax | perl -nE '/-s rabbit boot/ && /^\s*(\d+).*?-A (\d+)/ && say "$2"'
96

Controller:

root@node-1:~# nproc
6
root@node-1:~# ps ax | perl -nE '/beam.*-sname rabbit/ && /^\s*(\d+).*?-A (\d+)/ && say "$2"'
96

[Fuel Devops McRobotson (fuel-devops-robot)]

Change restored by Dmitry Mescheryakov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/19143

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Igor Shishkin <email address hidden> on branch: master
Review: https://review.fuel-infra.org/19143

[Fuel Devops McRobotson (fuel-devops-robot)]

Change restored by Igor Shishkin <email address hidden> on branch: master
Review: https://review.fuel-infra.org/19143

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19143
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: 1a66e660e0d3672bfc4a8e72cfa1f642be866056
Author: Alexey Lebedeff <email address hidden>
Date: Mon Jun 6 17:18:58 2016

Autotune erlang thread pool size

Backport from upstream https://github.com/binarin/rabbitmq-server/tree/rabbitmq-server-151

Other patches are moved around and added to streamline centos and ubuntu packaging.

Change-Id: I669523ee1db5e1d30af247dd94e36cd31613c4a9
Closes-Bug: 1565829