[plugin][rabbitmq] add output of: rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'

[Felipe Reyes]

[Impact]

We have seen customer cases where the output of
`rabbit_diagnostics:maybe_stuck()` provided helpful information in
debugging the root cause. Automatically collecting this information
will potentially expedite the debugging of rabbitmq-server related
issues.

[Test Case]

Run the rabbitmq plugin on a machine running a rabbitmq broker. On a
fresh Focal install, the output will look like the following:

    2020-09-18 12:55:47 There are 257 processes.
    2020-09-18 12:55:47 Investigated 1 processes this round, 5000ms to go.
    2020-09-18 12:55:47 Investigated 1 processes this round, 4500ms to go.
    2020-09-18 12:55:48 Investigated 1 processes this round, 4000ms to go.
    2020-09-18 12:55:48 Investigated 1 processes this round, 3500ms to go.
    2020-09-18 12:55:49 Investigated 1 processes this round, 3000ms to go.
    2020-09-18 12:55:49 Investigated 1 processes this round, 2500ms to go.
    2020-09-18 12:55:50 Investigated 1 processes this round, 2000ms to go.
    2020-09-18 12:55:50 Investigated 1 processes this round, 1500ms to go.
    2020-09-18 12:55:51 Investigated 1 processes this round, 1000ms to go.
    2020-09-18 12:55:51 Investigated 1 processes this round, 500ms to go.
    2020-09-18 12:55:52 Found 1 suspicious processes.
    2020-09-18 12:55:52 [{pid,<10643.1.0>},
                         {registered_name,erts_code_purger},
                         {current_stacktrace,
                             [{erts_code_purger,wait_for_request,0,[]}]},
                         {initial_call,{erts_code_purger,start,0}},
                         {message_queue_len,0},
                         {links,[]},
                         {monitors,[]},
                         {monitored_by,[]},
                         {heap_size,1598}]
    ok

In case some Erlang runtime threads are stuck they will show in the
above list.

[Regression Potential]

This function was introduced with rabbitmq-server-3.3.4 which covers
Xenial upwards. The command would not work on earlier versions of
rabbitmq-server.

[Other Info]

* Upstream commits:
https://github.com/sosreport/sos/commit/af5891786ff8c4e33c341fe9cca690fc77ac768f
https://github.com/sosreport/sos/commit/5b5ebe2d2c9d579931c8d6a78101e16d6abfa527

[Original Description]

Extend the rabbitmq plugin to include the output of:

    sudo rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'

This information is useful to get insights of the state of the erlang
virtual machine.

[Nicolas Bock (nicolasbock)]

Pushed: https://github.com/sosreport/sos/pull/2189

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.0-1ubuntu1

---------------
sosreport (4.0-1ubuntu1) groovy; urgency=medium

  [Eric Desrochers]
  * d/p/0003-sosclean-fix-handling-of-filepath-with-archive-name.patch:
    - Fixes the splitting of filepaths within the archive,
      when the archive name is included in the filename
      inside the archive. (LP: #1896222)

  * d/p/0004-sosclean-fix-tarball-skipping-regex.patch:
    - Fix tarball skipping regex

  [Dan Hill]
  * d/p/0005-ceph-collect-balancer-and-pg-autoscale-status.patch:
    - Collect balancer and pg-autoscale status (LP: #1893109)

  [Nicolas Bock]
  * d/p/0006-rabbitmq-add-info-on-maybe-stuck-processes.patch:
    - Add information on maybe_stuck() processes for RMQ. (LP: #1890846)

 -- Eric Desrochers <email address hidden> Fri, 18 Sep 2020 09:23:04 -0400

[Eric Desrochers (slashd)]

@nicolas,

Could you please file the SRU template before we can proceed with the SRU ?

[Eric Desrochers (slashd)]

After a discussion between Nicolas and I, a new commit has been proposed to sosreport upstream:
https://github.com/sosreport/sos/pull/2239/commits/5b5ebe2d2c9d579931c8d6a78101e16d6abfa527

[Eric Desrochers (slashd)]

I'm pushing the commit into Groovy as we speak.

[Eric Desrochers (slashd)]

It is now found in groovy-releases:

---
sosreport (4.0-1ubuntu2) groovy; urgency=medium

  [Nicolas Bock]
  * d/p/0007-rabbitmq-add-10sec-timeout-to-call-to-maybestuck.patch:
    - Add 10 second timeout to call to `maybe_stuck()`.
---

[Łukasz Zemczak (sil2100)]

Hello Felipe, or anyone else affected,

Accepted sosreport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sosreport/4.0-1~ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nicolas Bock (nicolasbock)]

Verified on Focal

[Nicolas Bock (nicolasbock)]

- installed sosreport-4.0-1~ubuntu0.20.04.2 in daily focal container
- installed rabbitmq-server-3.8.2-0ubuntu1.1
- ran sosreport --only-plugins rabbitmq
- verfied that sos_commands/rabbitmq/rabbitmqctl│All snaps up to date.
_eval_rabbit_diagnostics_maybe_stuck exists and contains correct information

[Launchpad Janitor (janitor)]

This bug was fixed in the package sosreport - 4.0-1~ubuntu0.20.04.2

---------------
sosreport (4.0-1~ubuntu0.20.04.2) focal; urgency=medium

  [Eric Desrochers]
  * d/p/0003-sosclean-fix-handling-of-filepath-with-archive-name.patch:
    - Fixes the splitting of filepaths within the archive,
      when the archive name is included in the filename
      inside the archive. (LP: #1896222)

  * d/p/0004-sosclean-fix-tarball-skipping-regex.patch:
    - Fix tarball skipping regex

  [Dan Hill]
  * d/p/0005-ceph-collect-balancer-and-pg-autoscale-status.patch:
    - Collect balancer and pg-autoscale status (LP: #1893109)

  [Nicolas Bock]
  * d/p/0006-rabbitmq-add-info-on-maybe-stuck-processes.patch:
    - Add information on maybe_stuck() processes for RMQ. (LP: #1890846)

  * d/p/0007-rabbitmq-add-10sec-timeout-to-call-to-maybestuck.patch:
    - Add 10 second timeout to call to `maybe_stuck()`.

 -- Eric Desrochers <email address hidden> Wed, 30 Sep 2020 14:29:50 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for sosreport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.