[Ussuri][CentOS8][Scenario004][rabbitmq-server-3.8] do not work with rabbitmqctl status | grep -F "{rabbit,"

[yatin]

Ussuri CentOS8 scenario004 fails[1] while checking rabbitmqctl status, as it looks for grep -F "{rabbit,".

rabbitmqctl status | grep -F "{rabbit,"' is used to check status for rabbitmq, but with rabbitmq 3.8 this do not work as rabbitmqctl status returns different format now[2].

the status check command is defined in puppet-tripleo:-
[puppet-tripleo]$ grep -nri "grep -F"
manifests/profile/pacemaker/rabbitmq_bundle.pp:333: $check_command = 'rabbitmqctl status | grep -F "{rabbit,"'
manifests/profile/pacemaker/rabbitmq.pp:154: command => 'rabbitmqctl status | grep -F "{rabbit,"',

[1] https://logserver.rdoproject.org/29/25329/19/check/periodic-tripleo-ci-centos-8-scenario004-standalone-master/a065434/logs/undercloud/var/log/extra/podman/containers/rabbitmq_init_bundle/stdout.log.txt.gz

[2]
$ sudo podman exec -it rabbitmq-bundle-podman-0 rabbitmqctl status
Status of node rabbit@c8004 ...
Runtime

OS PID: 679
OS: Linux
Uptime (seconds): 97
RabbitMQ version: 3.8.2
Node name: rabbit@c8004
Erlang configuration: Erlang/OTP 22 [erts-10.5.6] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64] [hipe]
Erlang processes: 420 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 15

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
 * amqp_client
 * cowboy
 * cowlib

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@c8004

Config files

 * /etc/rabbitmq/rabbitmq.config

Log file(s)

 * /<email address hidden>
 * /var/log/rabbitmq/rabbit@c8004_upgrade.log

Alarms

(none)

Memory

Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 3.2768 gb
code: 0.0295 gb (29.72 %)
other_proc: 0.0281 gb (28.31 %)
other_system: 0.0133 gb (13.42 %)
allocated_unused: 0.0113 gb (11.38 %)
reserved_unallocated: 0.0111 gb (11.22 %)
other_ets: 0.003 gb (3.05 %)
atom: 0.0015 gb (1.53 %)
plugins: 0.0007 gb (0.7 %)
metrics: 0.0002 gb (0.21 %)
mgmt_db: 0.0002 gb (0.15 %)
binary: 0.0001 gb (0.15 %)
mnesia: 0.0001 gb (0.08 %)
quorum_ets: 0.0 gb (0.04 %)
msg_index: 0.0 gb (0.03 %)
connection_other: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 65439
Sockets: 0, limit: 58893

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 69.5478 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: 192.168.101.18, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 15672, protocol: http, purpose: HTTP API

Also --formatter option is introduced in rabbitmqctl status, where "erlang" format can be used to get similar behavior

$ sudo podman exec -it rabbitmq-bundle-podman-0 rabbitmqctl status --formatter erlang | grep -F "{rabbit,"
      {rabbit,"RabbitMQ","3.8.2"},

"rabbitmqctl status |grep -Fi pid" should work though for both, me not sure what's the better way to check this to get it work for both old and new versions.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/710233

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/710233
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=e60351ee09eed0217d52aa1bad80f8c02501f0fe
Submitter: Zuul
Branch: master

commit e60351ee09eed0217d52aa1bad80f8c02501f0fe
Author: Damien Ciabrini <email address hidden>
Date: Thu Feb 27 12:35:32 2020 +0100

    HA: fix rabbitmq readiness check for rabbitmq-server 3.8

    In HA profiles, we wait for rabbitmq application readiness by
    parsing the output of "rabbitmqctl status". This breaks with
    rabbitmq-server 3.8 which changed the output of that command.

    Fix our check by using a "rabbitmqctl eval" and by relying on
    a stable function call rather than parsing output. This
    approach works for rabbitmq-server 3.6 to 3.8.

    Change-Id: Id88d0aee74e4b26fd64bbc2da5d0c0fc4bbd6644
    Co-Authored-By: Yatin Karel <email address hidden>
    Closes-Bug: #1864962

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 12.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/740307

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740307
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=207e6d621e5055a62c557fb051621f34c96c8e42
Submitter: Zuul
Branch: stable/train

commit 207e6d621e5055a62c557fb051621f34c96c8e42
Author: Damien Ciabrini <email address hidden>
Date: Thu Feb 27 12:35:32 2020 +0100

    HA: fix rabbitmq readiness check for rabbitmq-server 3.8

    In HA profiles, we wait for rabbitmq application readiness by
    parsing the output of "rabbitmqctl status". This breaks with
    rabbitmq-server 3.8 which changed the output of that command.

    Fix our check by using a "rabbitmqctl eval" and by relying on
    a stable function call rather than parsing output. This
    approach works for rabbitmq-server 3.6 to 3.8.

    Change-Id: Id88d0aee74e4b26fd64bbc2da5d0c0fc4bbd6644
    Co-Authored-By: Yatin Karel <email address hidden>
    Closes-Bug: #1864962
    (cherry picked from commit e60351ee09eed0217d52aa1bad80f8c02501f0fe)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 11.5.0 release.