Pacemaker logs contain errors from 'umm' utility: 'p_rabbitmq-server_start_0:22168:stderr [ /usr/bin/umm: line 40: runlevel: command not found ]'

[Artem Panchenko]

While debugging an issue with RabbitMQ server I noticed that pacemaker logs contains a lot of such lines:

fgrep umm /var/log/pacemaker.log | tail -3
Dec 22 20:31:37 [18915] node-1.test.domain.local pacemaker_remoted: notice: operation_finished: p_rabbitmq-server_monitor_27000:8182:stderr [ /usr/bin/umm: line 40: runlevel: command not found ]
Dec 22 20:31:37 [18915] node-1.test.domain.local pacemaker_remoted: notice: operation_finished: p_rabbitmq-server_monitor_27000:8182:stderr [ /usr/bin/umm: line 40: runlevel: command not found ]
Dec 22 20:31:37 [18915] node-1.test.domain.local pacemaker_remoted: notice: operation_finished: p_rabbitmq-server_monitor_27000:8182:stderr [ /usr/bin/umm: line 40: runlevel: command not found ]
fgrep -c umm /var/log/pacemaker.log
5046

There is a script which is executed when BASH profile is loaded on nodes:

root@node-1:~# cat /etc/profile.d/umm.sh
[ "z`umm status`" == "zumm" ] && cat /etc/issue.mm

And each time ocf script executes commands as 'rabbitmq' user [0] it gets an error in stderr from bash, like this one:

root@node-1:~# su - rabbitmq -s /bin/bash -c 'whoami'
/usr/bin/umm: line 40: runlevel: command not found
rabbitmq

It happens because 'runlevel' binary is not in PATH:

root@node-1:~# which runlevel
/sbin/runlevel
root@node-1:~# su - rabbitmq -s /bin/bash -c 'echo $PATH'
/usr/bin/umm: line 40: runlevel: command not found
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

Steps to reproduce:

1. Deploy environment
2. Login to controller node via SSH
3. Execute command `su - rabbitmq -s /bin/bash -c 'whoami'`

Expected result: it returns "rabbitmq" in stdout, stderr is empty

Actual: stderr contains error

[0] https://github.com/openstack/fuel-library/blob/master/files/fuel-ha-utils/ocf/rabbitmq#L322-L323

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/260712

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/260712
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ead206497dfe60eb591ce2acca15fc85c1b57861
Submitter: Jenkins
Branch: master

commit ead206497dfe60eb591ce2acca15fc85c1b57861
Author: Artem Panchenko <email address hidden>
Date: Tue Dec 22 22:52:15 2015 +0200

    Use absolute path to runlevel binary in umm

    Change-Id: Idcde2e87eb14089e4a152f5936d0ea197faa0793
    Closes-bug: #1528686

[Alexandr Kostrikov (akostrikov-mirantis)]

Verified on 9.0 RC2(490)

root@node-1:~# pcs status| grep rabbit
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
root@node-1:~# cat /etc/profile.d/umm.sh
if [ "z`umm status`" = "zumm" ] ; then
    cat /etc/issue.mm
    [ "$#" != "0" ] && echo "$@"
fi
root@node-1:~# su - rabbitmq -s /bin/bash -c 'whoami'
rabbitmq
root@node-1:~#
root@node-1:~# which runlevel
/sbin/runlevel

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/352529

[Denis Meltsaykin (dmeltsaykin)]

Closing as Won't Fix for 8.0 as this is a low importance non-customer-found bug.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Guryanov (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/352529
Reason: this is a fix for low bug, no need to merge to 8.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/451362

[OpenStack Infra (hudson-openstack)]

Change abandoned by Roman Vyalov (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/451362

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/451417

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/451417
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c206327d51a799eca7d41f46b1cb235c34e18b30
Submitter: Jenkins
Branch: stable/8.0

commit c206327d51a799eca7d41f46b1cb235c34e18b30
Author: Artem Panchenko <email address hidden>
Date: Tue Dec 22 22:52:15 2015 +0200

    Use absolute path to runlevel binary in umm

    There is no PATH profile load in that script. So when it invoked by
    pacemaker we have to use abosolute path.

    Change-Id: I7eca95667c2711f9611c87ff58c02487dbdc019c
    Picked-From: Idcde2e87eb14089e4a152f5936d0ea197faa0793
    Closes-Bug: #1528686
    Related-Bug: #1558963

[TatyanaGladysheva (tgladysheva)]

Verified on 8.0 + MU4 updates.

Actual results:
root@node-1:~# su - rabbitmq -s /bin/bash -c 'whoami'
rabbitmq
root@node-1:~# fgrep -c umm /var/log/pacemaker.log
0