RabbitMQ OCF script ignores settings in /etc/default/rabbitmq-server

[Roman Podoliaka]

RabbitMQ OCF script ignores settings in /etc/default/rabbitmq-server when (re)starting the RabbitMQ process.

Given the fact that distribution packages increase the soft and hard limits for the number of open file descriptors by sourcing /etc/default/rabbitmq-server in the init scripts, this may lead to situations when RabbitMQ (re)started by OCF script will behave differently from the one (re)started by init-script, e.g. it will have a smaller limit for TCP sockets (the issue we ran into on the scale lab).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/144057

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144057
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=cc7842ceaee973edaa1c4bcfc0fdd4672e241bd5
Submitter: Jenkins
Branch: master

commit cc7842ceaee973edaa1c4bcfc0fdd4672e241bd5
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Dec 25 22:40:25 2014 +0100

    Set ulimit from /etc/default/rabbitmq-server

    Allow rabbitmq to handle a lot of connections on scale

    Change-Id: Ia63e9eae2344535a2b0213c0fb2e9693c6317735
    Closes-Bug: 1405657
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/144634

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/144737

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144634
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2ba6d3eb2da994c150fe01f785d7f47805380564
Submitter: Jenkins
Branch: stable/5.1

commit 2ba6d3eb2da994c150fe01f785d7f47805380564
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Dec 25 22:40:25 2014 +0100

    Set ulimit from /etc/default/rabbitmq-server

    Allow rabbitmq to handle a lot of connections on scale

    Change-Id: Ia63e9eae2344535a2b0213c0fb2e9693c6317735
    Closes-Bug: 1405657
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144737
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=220c9099a8a73ce46e65b32aae476f2494dc146c
Submitter: Jenkins
Branch: stable/6.0

commit 220c9099a8a73ce46e65b32aae476f2494dc146c
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Dec 25 22:40:25 2014 +0100

    Set ulimit from /etc/default/rabbitmq-server

    Allow rabbitmq to handle a lot of connections on scale

    Change-Id: Ia63e9eae2344535a2b0213c0fb2e9693c6317735
    Closes-Bug: 1405657
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[Roman Podoliaka (rpodolyaka)]

Apparently, this hasn't been fixed by the change to OCF script. This has been reproduced on the scale lab again:

http://xsnippet.org/360417/raw/

http://xsnippet.org/360418/raw/

[Roman Podoliaka (rpodolyaka)]

I wonder if we should just set the ulimit for rabbitmq user in policy.d instead

[Roman Podoliaka (rpodolyaka)]

Heh, is it me or we missed that $OCF_RESKEY_binary will contain *full* path to rabbitmq binary and thus /etc/default/${OCF_RESKEY_binary} will make no sense?

http://xsnippet.org/360419/raw/

[Bogdan Dobrelya (bogdando)]

I cannot reproduse the issue with 6.1:
[root@node-1 fuel-library]# cat /etc/default/rabbitmq-server
# This file is sourced by /etc/init.d/rabbitmq-server. Its primary
# reason for existing is to allow adjustment of system limits for the
# rabbitmq-server process.
#
# Maximum number of open file handles. This will need to be increased
# to handle many simultaneous connections. Refer to the system
# documentation for ulimit (in man bash) for more information.
#
ulimit -H -n 105472
ulimit -S -n 102400
[root@node-1 fuel-library]# su rabbitmq -s /bin/sh -c "ulimit -n"
102400
[root@node-1 fuel-library]# grep files /proc/$(pidof beam.smp)/limits
Max open files 102400 112640 files

[Bogdan Dobrelya (bogdando)]

And rabbitmqctl status from the same env:
...
 {file_descriptors,
     [{total_limit,102300},
      {total_used,3},
      {sockets_limit,92068},
      {sockets_used,1}]},
 {processes,[{limit,1048576},{used,356}]},

[Bogdan Dobrelya (bogdando)]

But you are right, Roman - /etc/default/${OCF_RESKEY_binary} will result as /etc/default//usr/sbin/rabbitmq-server and should be fixed as well

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/147099

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/147107

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/147115

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/147116

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/5.1
Review: https://review.openstack.org/147107

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/147115

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147099
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=fc406a52917a9947e213ea8b1b98fc186fb56554
Submitter: Jenkins
Branch: master

commit fc406a52917a9947e213ea8b1b98fc186fb56554
Author: Sergii Golovatiuk <email address hidden>
Date: Wed Jan 14 10:55:27 2015 +0100

    Specify static defaults rather than variable

    * Specify static file name for ulimits rather than using OCF_OCF_RESKEY_binary
      as it may be specified with full path
    * Move OCF resource from openstack to claster resource for consistancy
      as it contains most of our OCF scripts

    Change-Id: Ibe740d8aa1d0613997868194fe7154bda145f55b
    Closes-Bug: 1405657
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147107
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=542909f515154bb107b93ead159641c65f29c5bf
Submitter: Jenkins
Branch: stable/5.1

commit 542909f515154bb107b93ead159641c65f29c5bf
Author: Sergii Golovatiuk <email address hidden>
Date: Wed Jan 14 11:51:25 2015 +0100

    Specify static defaults rather than variable

    * Specify static file name for ulimits rather than using OCF_OCF_RESKEY_binary
      as it may be specified with full path

    Change-Id: Ibe740d8aa1d0613997868194fe7154bda145f55b
    Closes-Bug: 1405657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147116
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=827079a97bfd8ebcb58af241ccf3fbaf459953a9
Submitter: Jenkins
Branch: stable/6.0

commit 827079a97bfd8ebcb58af241ccf3fbaf459953a9
Author: Sergii Golovatiuk <email address hidden>
Date: Wed Jan 14 12:09:53 2015 +0100

    Specify static defaults rather than variable

    * Specify static file name for ulimits rather than using OCF_OCF_RESKEY_binary
      as it may be specified with full path

    Change-Id: Ibe740d8aa1d0613997868194fe7154bda145f55b
    Closes-Bug: 1405657
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[Anastasia Palkina (apalkina)]

Verified on ISO #92

"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"

1. Kill rabbitmq on one of controllers
2. Wait until rabbitmq restart
3.
[root@node-6 ~]# ps aux | grep rabbit
root 5759 0.0 0.0 11300 940 ? S 10:22 0:00 /bin/sh /usr/sbin/rabbitmq-server
root 5775 0.0 0.0 48572 1176 ? S 10:22 0:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 5776 11.8 3.6 1165476 108084 ? Ssl 10:22 0:16 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -sname rabbit@node-6 -boot start_sasl -config /etc/rabbitmq/rabbitmq -rabbit tcp_listeners [{"auto",5673}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@node-6-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@node-6"
rabbitmq 6057 0.0 0.0 10792 532 ? Ss 10:22 0:00 inet_gethost 4
rabbitmq 6060 0.0 0.0 12896 672 ? S 10:22 0:00 inet_gethost 4
root 12623 0.0 0.0 103248 884 pts/0 S+ 10:24 0:00 grep rabbit

4. See limits for needed processes

[root@node-6 ~]# cat /proc/5759/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes ...