False positives in reporting server unavailable

[Robbert Muller]

[Impact]

 * If a rabbitmq server has no socket descriptors used the code
   triggers a false positive and errors out mistakenly reporting the
   server in a bad state.

 * Backport the fix from upstream, which is there for more than 3 years
   but hasn't tagged a new release since then.

[Test Case]

$ apt install rabbitmq-server nagios-plugins-rabbitmq
# Enable management http api
$ rabbitmq-plugins enable rabbitmq_management
# Now do the same via the nagios plugin (no actual nagios server needed)
$ cd /usr/lib/nagios/plugins-rabbitmq
$ ./check_rabbitmq_server --node $(hostname -s) --hostname 127.0.0.1 --port=15672 -u "guest" -p "guest"

Output bad case:
RABBITMQ_SERVER CRITICAL - Unable to get values for Sockets | Memory=0.57%;80;90 Process=0.04%;80;90 FD=0.05%;80;90
Output good case:
RABBITMQ_SERVER OK - Memory OK (0.57%) Process OK (0.04%) FD OK (0.05%) Sockets OK (0.00%) | Memory=0.57%;80;90 Process=0.04%;80;90 FD=0.05%;80;90 Sockets=0.00%;80;90

[Regression Potential]

 * The change is fortunately minimal and therefore easy to review.
   The old check by accident caused a value of 0 to become true due to the
   negation. Actually the check should have been if the value is defined
   at all. The regression impacts of this change IMHO are low, I can think
   of a scenario where people got used to see all idle servers in this
   particular broken state which will after the fix be reported "green"
   now - but I don't think that is something that should stop the SRU.

[Other Info]

 * This is harder to test on Xenial as the default setup there will not
   have zero socket descriptors, but the problem is the same.

----

On a unused rabbitmq machine, the server monitoring reports that it was unable to get a value for sockets.

This is not true, except the value is 0. In upstream this is already fixed by replacing !$used by !defined($used) on line 168 in /usr/lib/nagios/plugins-rabbitmq/check_rabbitmq_server

Upstream: https://github.com/nagios-plugins-rabbitmq/nagios-plugins-rabbitmq/blob/master/scripts/check_rabbitmq_server#L191

Description: Ubuntu 16.04.3 LTS
Release: 16.04

nagios-plugins-rabbitmq:
  Installed: 1:1.2.0-2.1
  Candidate: 1:1.2.0-2.1
  Version table:
 *** 1:1.2.0-2.1 500
        500 http://nl.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://nl.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
        100 /var/lib/dpkg/status

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Fixes issue" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Christian Ehrhardt  (paelzer)]

The actual upstream issue is:
https://github.com/nagios-plugins-rabbitmq/nagios-plugins-rabbitmq/issues/57
With the fix being:
https://github.com/nagios-plugins-rabbitmq/nagios-plugins-rabbitmq/commit/caabebdf4e1d820c19b7d7fa8e682848f22834e9

Versions in Ubuntu are:
 nagios-plugins-rabbitmq | 1:1.2.0-2.1 | xenial/universe | source, all
 nagios-plugins-rabbitmq | 1:1.2.0-2.2 | bionic/universe | source, all
 nagios-plugins-rabbitmq | 1:1.2.0-2.2 | disco/universe | source, all
 nagios-plugins-rabbitmq | 1:1.2.0-2.2 | eoan/universe | source, all

But even though we are slightly outdated (2.0.2 vs 2.0.3) there was no release since 2.0.3 (2016-04-14) which means there isn't any official release including this yet.

[Christian Ehrhardt  (paelzer)]

After some debugging I came up with a simple Testcase for this issue:

$ apt install rabbitmq-server nagios-plugins-rabbitmq
# Enable management http api
$ rabbitmq-plugins enable rabbitmq_management
# Now do the same via the nagios plugin (no actual nagios server needed)
$ cd /usr/lib/nagios/plugins-rabbitmq
$ ./check_rabbitmq_server --node $(hostname -s) --hostname 127.0.0.1 --port=15672 -u "guest" -p "guest"

Output bad case:
RABBITMQ_SERVER CRITICAL - Unable to get values for Sockets | Memory=0.57%;80;90 Process=0.04%;80;90 FD=0.05%;80;90
Output good case:
RABBITMQ_SERVER OK - Memory OK (0.57%) Process OK (0.04%) FD OK (0.05%) Sockets OK (0.00%) | Memory=0.57%;80;90 Process=0.04%;80;90 FD=0.05%;80;90 Sockets=0.00%;80;90

[Christian Ehrhardt  (paelzer)]

Optional checks if something isn't working
Note: some of the commands only exist in newer versions of rabbitmq
# Check that rabbit is up and "happy"
# rabbitmqctl -n rabbit@$(hostname -s) node_health_check
Timeout: 70 seconds ...
Checking health of node rabbit@e ...
Health check passed

# Check enabled management http api after enabling
$ rabbitmq-plugins list

# Check the API to work with rabbitmqadmin
rabbitmqadmin show overview --port=15672 --node $(hostname -s)

[Christian Ehrhardt  (paelzer)]

Per the test above Bionic/Disco/Eoan are affected.

The default install of Xenial is not, it has the same issue in the code but it seems the default setup has something not triggering this. Even when copying in the same version from B/D/E that fails there it works. So it must be like "there are sockets" in the older rabbitmq default install.

[Christian Ehrhardt  (paelzer)]

In fact on Xenial the default setup as one can check via:
$ rabbitmqctl add_user full_access secret
Creating user "full_access"
$ rabbitmqctl set_user_tags full_access administrator
Setting tags for user "full_access" to [administrator]
# web browse to <IP>:15672 then and we will find 1 socket on the xenial install used, but 0 on the other releases

But there is no connection open (as usually you'd close the connection to free up socket descriptors).

Checking a remote server that has zero sockets for now has other issues (timeouts).
But since we know the same bug is in all lets fix Xenial as well, even if it is harder to recreate there.

[Christian Ehrhardt  (paelzer)]

This is a fix that does not touch the current Freeze being active in Eoan, lets prepare a fix for this and verify it.

Note: I have already locally modified the script and know the fix works. But to be sure we don#t run into issues on re-build lets do the full path.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nagios-plugins-rabbitmq - 1:1.2.0-2.2ubuntu1

---------------
nagios-plugins-rabbitmq (1:1.2.0-2.2ubuntu1) eoan; urgency=medium

  * Avoid breaking to get stats from an idle server that has
    no socket descriptors open (LP: #1787110)

 -- Christian Ehrhardt <email address hidden> Tue, 24 Sep 2019 10:54:04 +0200

[Christian Ehrhardt  (paelzer)]

The fix in Eoan works as expected.
Uploaded the same fix (as tested in containers already) to X/B/D for SRU review.

[Brian Murray (brian-murray)]

Hello Robbert, or anyone else affected,

Accepted nagios-plugins-rabbitmq into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nagios-plugins-rabbitmq/1:1.2.0-2.2ubuntu0.19.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Robbert, or anyone else affected,

Accepted nagios-plugins-rabbitmq into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nagios-plugins-rabbitmq/1:1.2.0-2.2ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Robbert, or anyone else affected,

Accepted nagios-plugins-rabbitmq into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nagios-plugins-rabbitmq/1:1.2.0-2.1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

Upgrading worked as expected:
# apt install nagios-plugins-rabbitmq
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  nagios-plugins-rabbitmq
1 upgraded, 0 newly installed, 0 to remove and 31 not upgraded.
Need to get 12.0 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu disco-proposed/universe amd64 nagios-plugins-rabbitmq all 1:1.2.0-2.2ubuntu0.19.04.1 [12.0 kB]
Fetched 12.0 kB in 0s (257 kB/s)
(Reading database ... 68294 files and directories currently installed.)
Preparing to unpack .../nagios-plugins-rabbitmq_1%3a1.2.0-2.2ubuntu0.19.04.1_all.deb ...
Unpacking nagios-plugins-rabbitmq (1:1.2.0-2.2ubuntu0.19.04.1) over (1:1.2.0-2.2) ...
Setting up nagios-plugins-rabbitmq (1:1.2.0-2.2ubuntu0.19.04.1) ...

On Bionic and Xenial it worked just the same ...

After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 nagios-plugins-rabbitmq all 1:1.2.0-2.2ubuntu0.18.04.1 [12.0 kB]
Fetched 12.0 kB in 0s (217 kB/s)
(Reading database ... 84378 files and directories currently installed.)
Preparing to unpack .../nagios-plugins-rabbitmq_1%3a1.2.0-2.2ubuntu0.18.04.1_all.deb ...
Unpacking nagios-plugins-rabbitmq (1:1.2.0-2.2ubuntu0.18.04.1) over (1:1.2.0-2.2) ...
Setting up nagios-plugins-rabbitmq (1:1.2.0-2.2ubuntu0.18.04.1) ...

After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/universe amd64 nagios-plugins-rabbitmq all 1:1.2.0-2.1ubuntu0.1 [11.9 kB]
Fetched 11.9 kB in 0s (176 kB/s)
(Reading database ... 36386 files and directories currently installed.)
Preparing to unpack .../nagios-plugins-rabbitmq_1%3a1.2.0-2.1ubuntu0.1_all.deb ...
Unpacking nagios-plugins-rabbitmq (1:1.2.0-2.1ubuntu0.1) over (1:1.2.0-2.1) ...
Setting up nagios-plugins-rabbitmq (1:1.2.0-2.1ubuntu0.1) ...

[Christian Ehrhardt  (paelzer)]

In ensured pre-upgrade the systems were affected and then:

B:
root@b:/usr/lib/nagios/plugins-rabbitmq# ./check_rabbitmq_server --node $(hostname -s) --hostname 127.0.0.1 --port=15672 -u "guest" -p "guest"
RABBITMQ_SERVER OK - Memory OK (0.45%) Process OK (0.03%) FD OK (0.04%) Sockets OK (0.00%) | Memory=0.45%;80;90 Process=0.03%;80;90 FD=0.04%;80;90 Sockets=0.00%;80;90

D:
root@d:/usr/lib/nagios/plugins-rabbitmq# ./check_rabbitmq_server --node $(hostname -s) --hostname 127.0.0.1 --port=15672 -u "guest" -p "guest"
RABBITMQ_SERVER OK - Memory OK (0.53%) Process OK (0.04%) FD OK (0.05%) Sockets OK (0.00%) | Memory=0.53%;80;90 Process=0.04%;80;90 FD=0.05%;80;90 Sockets=0.00%;80;90

X is and stays ok, but as outlined there I couldn't get the socket count down to zero (which would trigger the former bug). But I tested to probe a remote which now worked as well as not seeing other issues in the scripts provided by nagios-plugins-rabbitmq (we haven't touched the others, so that was expected)

Marking verified

[Launchpad Janitor (janitor)]

This bug was fixed in the package nagios-plugins-rabbitmq - 1:1.2.0-2.2ubuntu0.19.04.1

---------------
nagios-plugins-rabbitmq (1:1.2.0-2.2ubuntu0.19.04.1) disco; urgency=medium

  * Avoid breaking to get stats from an idle server that has
    no socket descriptors open (LP: #1787110)

 -- Christian Ehrhardt <email address hidden> Tue, 24 Sep 2019 10:54:04 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for nagios-plugins-rabbitmq has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nagios-plugins-rabbitmq - 1:1.2.0-2.2ubuntu0.18.04.1

---------------
nagios-plugins-rabbitmq (1:1.2.0-2.2ubuntu0.18.04.1) bionic; urgency=medium

  * Avoid breaking to get stats from an idle server that has
    no socket descriptors open (LP: #1787110)

 -- Christian Ehrhardt <email address hidden> Tue, 24 Sep 2019 10:54:04 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package nagios-plugins-rabbitmq - 1:1.2.0-2.1ubuntu0.1

---------------
nagios-plugins-rabbitmq (1:1.2.0-2.1ubuntu0.1) xenial; urgency=medium

  * Avoid breaking to get stats from an idle server that has
    no socket descriptors open (LP: #1787110)

 -- Christian Ehrhardt <email address hidden> Tue, 24 Sep 2019 10:54:04 +0200