Failure to get free disk space breaks "rabbitmqctl status" command

Bug #1999816 reported by Jorge Merlino
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rabbitmq-server (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Medium
Jorge Merlino
Jammy
Fix Released
Medium
Jorge Merlino
Kinetic
Fix Released
Medium
Jorge Merlino

Bug Description

[Impact]

When for some reason the df command fails to get the disk free space (for example timeout on a heavily loaded system) the result is a harcoded value of "unknown". As this is not a valid number this generates arithmetic errors when the "rabbitmqctl status" command is run and tries to divide that value to convert it to another unit.

This has been fixed upstream here:
https://github.com/rabbitmq/rabbitmq-server/pull/4897

[Test Plan]

The df command can be linked to another file that just waits for a few minutes to force a timeout for example: [detailed steps in comment #5].

#!/bin/bash
sleep 5m

After the timeout occurs the "rabbitmqctl status" returns an error with the unpatched version. After the patch it shows all the information and displays unknown in the free space line.

[Where problems could occur]

The patch just changes the display of information, it should not break anything in the core operations of the package

Changed in rabbitmq-server (Ubuntu):
assignee: nobody → Jorge Merlino (jorge-merlino)
description: updated
description: updated
Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Patch for jammy

tags: added: sts
tags: added: sts-sponsors ubuntu-sponsors
Changed in rabbitmq-server (Ubuntu):
status: New → In Progress
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1999816_jammy.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Patch for focal

Changed in rabbitmq-server (Ubuntu Focal):
assignee: nobody → Jorge Merlino (jorge-merlino)
Changed in rabbitmq-server (Ubuntu Jammy):
assignee: nobody → Jorge Merlino (jorge-merlino)
description: updated
Changed in rabbitmq-server (Ubuntu Focal):
status: New → In Progress
Changed in rabbitmq-server (Ubuntu Jammy):
status: New → In Progress
tags: added: se-sru-sponsor-mfo
removed: sts-sponsors ubuntu-sponsors
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (4.9 KiB)

Hey Jorge,

Thanks for the bug report, SRU template, and debdiffs!

I just reviewed those; please find the comments below.

I'll keep the points detailed in case they might help
other readers preparing for submitting SRUs in Ubuntu.

Thanks!
Mauricio

...

SRU template:
---

- Added [Impact] section title [0].
- Added more steps/commands to clarify the reproducer.

Fix:
---

- Added nomination for kinetic (please attach debdiff).

It's still supported for ~3 months :)

$ ubuntu-distro-info --series=kinetic --days=eol
174

...

Checking the provided link for fix commit title.

https://github.com/rabbitmq/rabbitmq-server/pull/4897

Checking the versions the commit was introduced,
and the stable branches with backports (nice!).

$ git clone https://github.com/rabbitmq/rabbitmq-server.git
$ cd rabbitmq-server

$ git log --oneline -G 'when is_number' origin/main -- deps/rabbitmq_cli/lib/rabbitmq/cli/information_unit.ex
9dc9b92a94f3 Fix `rabbitmqctl status` when the disk_free cannot be determined

$ git describe --contains 9dc9b92a94f3
v3.11.0-beta.1~288^2~1

$ rmadison -a source rabbitmq-server
 rabbitmq-server | 3.2.4-1 | trusty | source
 rabbitmq-server | 3.2.4-1ubuntu0.1 | trusty-security | source
 rabbitmq-server | 3.2.4-1ubuntu0.1 | trusty-updates | source
 rabbitmq-server | 3.5.7-1 | xenial | source
 rabbitmq-server | 3.5.7-1ubuntu0.16.04.2 | xenial-security | source
 rabbitmq-server | 3.5.7-1ubuntu0.16.04.4 | xenial-updates | source
 rabbitmq-server | 3.6.10-1 | bionic | source
 rabbitmq-server | 3.6.10-1ubuntu0.5 | bionic-security | source
 rabbitmq-server | 3.6.10-1ubuntu0.5 | bionic-updates | source
 rabbitmq-server | 3.8.2-0ubuntu1~ubuntu18.04.1 | bionic-backports | source
 rabbitmq-server | 3.8.2-0ubuntu1 | focal | source
 rabbitmq-server | 3.8.2-0ubuntu1.3 | focal-security | source
 rabbitmq-server | 3.8.2-0ubuntu1.3 | focal-updates | source
 rabbitmq-server | 3.9.13-1 | jammy | source
 rabbitmq-server | 3.9.13-1 | kinetic | source
 rabbitmq-server | 3.10.8-1.1 | lunar | source

1) lunar: not needed.

$ git log --oneline -G 'when is_number' origin/v3.10.x -- deps/rabbitmq_cli/lib/rabbitmq/cli/information_unit.ex
abb059610eac Fix `rabbitmqctl status` when the disk_free cannot be determined
$ git describe --contains abb059610eac
v3.10.3~19^2~1

2) kinetic/jammy: needed. [kinetic not nominated/debdiff]

$ git log --oneline -G 'when is_number' origin/v3.9.x -- deps/rabbitmq_cli/lib/rabbitmq/cli/information_unit.ex
17c9f6960724 Fix `rabbitmqctl status` when the disk_free cannot be determined
$ git describe --contains 17c9f6960724
v3.9.19~10^2~1

3) focal: needed

$ git log --oneline -G 'when is_number' origin/v3.8.x -- deps/rabbitmq_cli/lib/rabbitmq/cli/information_unit.ex
7a5a467d4be4 Fix `rabbitmqctl status` when the disk_free cannot be determined
$ git describe --contains 7a5a467d4be4
v3.8.33~2^2~1

1.1) lunar: double check

$ pull-lp-sourc...

Read more...

Changed in rabbitmq-server (Ubuntu Kinetic):
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Jorge Merlino (jorge-merlino)
status: Confirmed → Triaged
Changed in rabbitmq-server (Ubuntu Jammy):
status: In Progress → Triaged
importance: Undecided → Medium
Changed in rabbitmq-server (Ubuntu Focal):
status: In Progress → Triaged
importance: Undecided → Medium
Changed in rabbitmq-server (Ubuntu):
status: In Progress → Fix Released
assignee: Jorge Merlino (jorge-merlino) → nobody
description: updated
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Steps to Reproduce
---

lxc launch ubuntu:focal rmq-focal
lxc exec rmq-focal -- su - ubuntu

sudo apt install --yes rabbitmq-server

$ sudo rabbitmqctl status | grep -i disk
Free Disk Space
Low free disk space watermark: 0.05 gb
Free disk space: 327.7867 gb

DF=$(which df)
SH=/tmp/df.sh

cat <<EOF >$SH
#!/bin/sh
sleep 5m
EOF

sudo chmod +x $SH
sudo mount --bind $SH $DF

$ sudo rabbitmqctl status
Status of node rabbit@rmq-focal ...
<... DELAY ...>
Error:
{{:unparseable, []}, {:gen_server, :call, [:rabbit_disk_monitor, :get_disk_free_limit, :infinity]}}

sudo kill $(pidof sleep)
sudo umount $DF
rm $SH

For doc purposes, the chain from rabbitmq-server to the df (sleep) command:

$ ps axf | sed -n '/rabbitmq-server$/,/sleep/p'
   1973 ? Ss 0:00 /bin/sh /usr/sbin/rabbitmq-server
   1977 ? Sl 0:31 \_ /usr/lib/erlang/erts-10.6.4/bin/beam.smp ...
   2244 ? Ss 0:00 \_ erl_child_setup 65536
   2273 ? Ss 0:00 \_ inet_gethost 4
   2274 ? S 0:00 | \_ inet_gethost 4
   4237 ? Ss 0:00 \_ /bin/sh -s unix:cmd
   4238 ? S 0:00 \_ /bin/sh /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@rmq-focal
   4239 ? S 0:00 \_ sleep 5m

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Updated patch for Focal

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Updated patch for Jammy

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Patch for Kinetic

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hi Jorge,

Thanks for the new debdiffs!

They look good; I just adjusted a few things:
- k/j: version: missing .1 suffix per [1] (on me)
- k/j/f: shorter 'd/p/' on changelog (optional)
- k/j/f: origin field pointing to commit vs. PR.
- f: origin keyword of 'backport' vs. 'upstream'
  (the backport matches the upstream's in v3.8.x).

The packages built correctly on PPAs with -updates
and -proposed enabled, and I quickly tested Focal.

$ sudo rabbitmqctl status | grep -i disk
Free Disk Space
Low free disk space watermark: 0.05 gb
Free disk space: undefined gb

The upgrade path across pockets/releases is good:

$ dpkg --compare-versions 3.8.2-0ubuntu1.3 lt 3.8.2-0ubuntu1.4 ; echo $?
0
$ dpkg --compare-versions 3.8.2-0ubuntu1.4 lt 3.9.13-1 ; echo $?
0
$ dpkg --compare-versions 3.9.13-1 lt 3.9.13-1ubuntu0.22.04.1 ; echo $?
0
$ dpkg --compare-versions 3.9.13-1 lt 3.9.13-1ubuntu0.22.10.1 ; echo $?
0
$ dpkg --compare-versions 3.9.13-1ubuntu0.22.04.1 lt 3.9.13-1ubuntu0.22.10.1 ; echo $?
0
$ dpkg --compare-versions 3.9.13-1ubuntu0.22.10.1 lt 3.10.8-1.1 ; echo $?
0

Uploaded to Kinetic, Jammy, and Focal.

Thanks,
Mauricio

[1] https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging

Changed in rabbitmq-server (Ubuntu Kinetic):
status: Triaged → In Progress
Changed in rabbitmq-server (Ubuntu Jammy):
status: Triaged → In Progress
Changed in rabbitmq-server (Ubuntu Focal):
status: Triaged → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I noticed that the focal patch is slightly different. It uses `:undefined` instead of `:unknown` in information_unit.ex. That does not match what is committed[1] upstream in the v3.8.x branch, and it remains using `:unknown` in v3.8.x up to this day.

Could you please check if that's correct, and if yes, why are we different from upstream in the focal patch?

I don't know erlang, but does the different word also translate to a different output in the status command?

1. https://github.com/rabbitmq/rabbitmq-server/commit/7a5a467d4be4

Changed in rabbitmq-server (Ubuntu Focal):
status: In Progress → Incomplete
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hey Andreas,

Thanks for catching that; I can't recollect why I mentioned the backport matched the v3.8.x branch:
> (the backport matches the upstream's in v3.8.x).

Let's wait for Jorge's details on that delta.

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

I've been investigating this issue but I could not find the exact difference between 3.8 and 3.9 that accounts for that change.
The output of the status command will indeed be different between 3.8 and 3.9 in this case as in one will say that the free space is unknown and in the other that the free space is undefined. I don't think that is important as long as the command works. It would be easy anyway to equalize that just changing the return value of the function to :unknown
I talked to upstream but they could not pinpoint the difference for me and they are not accepting patches for 3.8 either as there will be no new versions.
So, to move this forward I propose modifying that patch to change the origin from upstream to backport and leave the reference for the original patch.
What do you think?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

What's not clear to me is why can't we use the upstream v3.8.x patch? And use :unknown, like all other branches?

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

The thing is that in 3.8 the convert function is called with a :undefined value when the disk space cannot not be obtained instead of a :unknown value. So, testing for an :unknown value has no effect. I could not find the the exact code that explain this difference.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

> I talked to upstream but they could not pinpoint the difference for me and they are not accepting patches for 3.8 either as there will be no new versions.

Did this conversation happen in an upstream github issue or somewhere else we can link to?

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

There are different conversations. I asked for pointers about the unknown/undefined problem here:

https://github.com/rabbitmq/rabbitmq-server/discussions/7210

The supported versions can be seen here:

https://www.rabbitmq.com/versions.html

And they previously rejected a PR to 3.8 in account of that:

https://github.com/rabbitmq/rabbitmq-server/pull/6671

Changed in rabbitmq-server (Ubuntu Focal):
status: Incomplete → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

> So, testing for an :unknown value has no effect. I could not find the the exact code that explain this difference.

You mean by keeping the upstream 3.8.x patch which uses "unknown", rabbitmqctl status would still crash in the end?

Here is the current chain of calls as I see it, please correct me if I'm wrong:

Without the fix:
- app APP1 calls "rabbitmqctl status", to get status
- rabbitmqctl status triggers some events that eventually call the `df` tool, which hangs/fails/whatever, rabbitmqctl gets something that is not a number, crashes
- APP1 notices that rabbitmqctl failed, reports that in some way, or even crashes itself

With the fix ("unknown"):
- app APP1 calls "rabbitmqctl status", to get status
- rabbitmqctl goes all the way down to getting df output, which fails in the same way, reports unknown, but now rabbitmqctl instead of crashing, just propagates that "unknown" value as the disk space, and does not crash
- APP1 gets status output, tries to check disk space, and now:
  - maybe it knows how to handle the fact that "unknown" is not a number, and behaves well
  - maybe it tries to parse "unknown" or "undefined" as a number, and crashes
  - maybe it tries to parse "unknown", but gets "undefined" instead, and crashes

I understand the fix for rabbitmqctl status not crashing, but that just makes it propagate the value that originally made it crash, to its caller (and there isn't really much else it can do). Do we know of any APP1 like in the above example? Is that something that we could test? Or should we wait and see if now something else (APP1) starts crashing, and then fix that, and so on?

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

That is a good question but in this case there is a human looking at the rabbitmqctl status who maybe is not even interested in the free space info (which can be easily obtained elsewhere) but can see no rabbitmq information on account of this bug.

So I don't know of any APP1 being affected by this result. Nevertheless I think It would be nice that the versions for focal and jammy return the same value when this error happens instead of using two different ones. I'll upload a new focal debdiff that also returns unknown to be consistent.

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Updated patch for Focal

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Thanks for the updated debdiff.

Note that this is still a backport, since it's not matching upstream commit https://github.com/rabbitmq/rabbitmq-server/commit/7a5a467d4be4,

Could you please change the Origin tag to mean:
a) it's a backport
b) use the https://github.com/rabbitmq/rabbitmq-server/commit/7a5a467d4be4 url for the commit, and not a pull request

And add some text to the description explaining why you made the change you made (essentially a summary of comment #18) above. Maybe prefix this explanation with something like "Ubuntu backport note:" or similar.

Thanks

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Updated patch for Focal

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hi Andreas and Jorge,

I reviewed the focal v4 debdiff (it does address the origin: tag as per #20 and #9, and adds change description), and also built it in a PPA and tested/confirmed it now prints 'unknown' (vs. 'undefined' in #9):

...
Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: unknown gb
...

Andreas, I'll upload it to Focal, as those seem to be the last pending items in your review (please reject it again if that's not the case).

Thanks!
Mauricio

tags: added: se-sponsor-mfo
removed: se-sru-sponsor-mfo
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Please test proposed package

Hello Jorge, or anyone else affected,

Accepted rabbitmq-server into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rabbitmq-server/3.9.13-1ubuntu0.22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in rabbitmq-server (Ubuntu Kinetic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-kinetic
Changed in rabbitmq-server (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Jorge, or anyone else affected,

Accepted rabbitmq-server into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rabbitmq-server/3.9.13-1ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in rabbitmq-server (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Jorge, or anyone else affected,

Accepted rabbitmq-server into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rabbitmq-server/3.8.2-0ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (rabbitmq-server/3.9.13-1ubuntu0.22.04.1)

All autopkgtests for the newly accepted rabbitmq-server (3.9.13-1ubuntu0.22.04.1) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

php-amqplib/3.1.2-1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#rabbitmq-server

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (rabbitmq-server/3.9.13-1ubuntu0.22.10.1)

All autopkgtests for the newly accepted rabbitmq-server (3.9.13-1ubuntu0.22.10.1) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

php-amqplib/3.2.0-1 (armhf, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#rabbitmq-server

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

DEP8 tests are green now, after a retry.

Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Tested version 3.8.2-0ubuntu1.4 on focal by redefining df and running rabbitmqctl status. It worked and returned:

Free disk space: unknown gb

so it is fine.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Tested version 3.9.13-1ubuntu0.22.04.1 on jammy by redefining df and running rabbitmqctl status. It worked and returned:

Free disk space: unknown gb

so it is fine.

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Tested version 3.9.13-1ubuntu0.22.10.1 on kinetic by redefining df and running rabbitmqctl status. It worked and returned:

Free disk space: unknown gb

so it is fine.

tags: added: verification-done-kinetic
removed: verification-needed-kinetic
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Jorge, could you please double-check the focal test? I couldn't get it to pass here with the updated package, and I tried twice in a different focal container:

ubuntu@f:~$ sudo apt install rabbitmq-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  rabbitmq-server
1 upgraded, 0 newly installed, 0 to remove and 52 not upgraded.
Need to get 11.5 MB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://br.archive.ubuntu.com/ubuntu focal-proposed/main amd64 rabbitmq-server all 3.8.2-0ubuntu1.4 [11.5 MB]
Fetched 11.5 MB in 1s (11.8 MB/s)
(Reading database ... 39116 files and directories currently installed.)
Preparing to unpack .../rabbitmq-server_3.8.2-0ubuntu1.4_all.deb ...
Unpacking rabbitmq-server (3.8.2-0ubuntu1.4) over (3.8.2-0ubuntu1.3) ...
Setting up rabbitmq-server (3.8.2-0ubuntu1.4) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.19) ...

ubuntu@f:~$ sudo rabbitmqctl status | grep -i disk
Free Disk Space
Low free disk space watermark: 0.05 gb
Free disk space: 143.2679 gb

ubuntu@f:~$ DF=$(which df)
ubuntu@f:~$ SH=/tmp/df.sh
ubuntu@f:~$
ubuntu@f:~$ cat <<EOF >$SH
> #!/bin/sh
> sleep 5m
> EOF
ubuntu@f:~$
ubuntu@f:~$ sudo chmod +x $SH
ubuntu@f:~$ sudo mount --bind $SH $DF

ubuntu@f:~$ sudo rabbitmqctl status
Status of node rabbit@f ...
Error:
{{:unparseable, []}, {:gen_server, :call, [:rabbit_disk_monitor, :get_disk_free_limit, :infinity]}}
ubuntu@f:~$

Jammy and kinetic passed fine. Focal is the one with the different patch...

tags: added: verification-needed-focal
removed: verification-done-focal
Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

I was able to reproduce that error in focal if I run the rabbitmqctl status command just after I change the df executable. I waited a few minutes in my tests. Can you confirm that it works fine after that first error?

I'll try to find out the reason for that ayway.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

In focal (haven't yet checked the others in this regard), rabbitmq (the server) seems to call "df" every 10s, regardless if I run rabbitmqctl status or not:

# execsnoop-bpfcc -T -n df
TIME PCOMM PID PPID RET ARGS
13:17:48 df 7203 7202 0 /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:17:48 df.orig 7204 7203 0 /usr/bin/df.orig -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:17:58 df 7209 7208 0 /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:17:58 df.orig 7210 7209 0 /usr/bin/df.orig -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:08 df 7212 7211 0 /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:08 df.orig 7213 7212 0 /usr/bin/df.orig -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:18 df 7215 7214 0 /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:18 df.orig 7216 7215 0 /usr/bin/df.orig -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:28 df 7218 7217 0 /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:28 df.orig 7219 7218 0 /usr/bin/df.orig -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:38 df 7221 7220 0 /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit
13:18:38 df.orig 7222 7221 0 /usr/bin/df.orig -kP /var/lib/rabbitmq/mnesia/rabbit@f-rabbit

(I used a /usr/bin/df wrapper that calls /usr/bin/df.orig "$@")

If you call rabbitmqctl-status in between those df calls, you will get the report from the last df run.

If I add the long sleep, and call rabbitmqctl status while that sleep is running, then my status command hangs until the sleep is over, or a timeout is reached.

How about changing the test case to have df exit without printing anything?

Like:
cat <<EOF >$SH
#!/bin/sh
exit 0
EOF

I noticed that in this case (focal at least) the server calls df once, probably notices it isn't working, and doesn't call it again, so no repeated calls every 10s. I left it for a while and it looks like the new frequency is every 2min. Once df is working again (if I let the wrapper call df.orig for example), then it resumes the 10s frequency.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I repeated the testing in all releases with this in mind, and am satisfied with the results.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions.

There is a focal SRU freeze ongoing at the moment, so I won't be releasing this for focal just yet.

There is no halted phasing on the previous update.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rabbitmq-server - 3.9.13-1ubuntu0.22.10.1

---------------
rabbitmq-server (3.9.13-1ubuntu0.22.10.1) kinetic; urgency=medium

  * d/p/lp1999816-fix-rabbitmqctl-status-disk-free-timeout.patch:
    Fix rabbitmqctl status when free disk space cannot be determined
    (LP: #1999816)

 -- Jorge Merlino <email address hidden> Mon, 30 Jan 2023 18:59:07 -0300

Changed in rabbitmq-server (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Update Released

The verification of the Stable Release Update for rabbitmq-server has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rabbitmq-server - 3.9.13-1ubuntu0.22.04.1

---------------
rabbitmq-server (3.9.13-1ubuntu0.22.04.1) jammy; urgency=medium

  * d/p/lp1999816-fix-rabbitmqctl-status-disk-free-timeout.patch:
    Fix rabbitmqctl status when free disk space cannot be determined
    (LP: #1999816).

 -- Jorge Merlino <email address hidden> Wed, 28 Dec 2022 21:34:18 -0300

Changed in rabbitmq-server (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Jorge Merlino (jorge-merlino) wrote :

Thank you for you work Andreas!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rabbitmq-server - 3.8.2-0ubuntu1.4

---------------
rabbitmq-server (3.8.2-0ubuntu1.4) focal; urgency=medium

  * d/p/lp1999816-fix-rabbitmqctl-status-disk-free-timeout.patch:
    Fix rabbitmqctl status when free disk space cannot be determined
    (LP: #1999816).

 -- Jorge Merlino <email address hidden> Wed, 22 Feb 2023 19:47:18 -0300

Changed in rabbitmq-server (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.