check_disk should look at /proc/mounts, not mtab

[Andrea Ieri]

We have a machine in production that suffered a failure in one of its drives. We attempted unmounting the failed drive, but that only partially succeeded: the failed drive is not mounted anymore, but the umount process is stuck in D state.

$ ps aux | grep umount
root 502476 0.0 0.0 10480 2108 pts/6 S+ 22:21 0:00 grep --color=auto umount
root 3879507 0.0 0.0 19000 1492 ? D Apr08 0:07 umount /srv/ceph/ceph1

With the broken drive gone, I have then removed the (now empty) mountpoint /srv/ceph/ceph1.

Now our usual check is alerting:

$ /usr/lib/nagios/plugins/check_disk -u GB -w 10% -c 4% -r 'node|ceph|udev' -i 'snap' -C -w 3% -c 1% -r 'instances|nova|libvirt|udev' -p /
DISK CRITICAL - /srv/ceph/ceph1 is not accessible: No such file or directory

I suspect this is due to check_disk running the -r regex against mtab[0], which umount hasn't had a chance to update yet.

Looking at /proc/mounts would better reflect the state of the system:

$ grep ceph1 /etc/mtab
/dev/bcache3 /srv/ceph/ceph1 xfs rw 0 0

$ grep ceph1 /proc/mounts
$

[0] https://github.com/monitoring-plugins/monitoring-plugins/blob/9661ee74885834f7b69ab0874c4e65bed0b871c9/gl/mountlist.c#L63

[Christian Ehrhardt  (paelzer)]

I thought /etc/mtab only exists for historical reasons.

And checking it shows:
lrwxrwxrwx 1 root root 19 Okt 12 2018 /etc/mtab -> ../proc/self/mounts

This is true at least back to Xenial (which I checked).
And for compatibility reasons programs often use /etc/mtab but it is actually the same.

Quoting the man page of mount:
"The programs mount and umount traditionally maintained a list of currently mounted filesystems in the file /etc/mtab. This real mtab file is still supported, but on current Linux systems it is better to make it a symlink to /proc/mounts instead, because a regular mtab file maintained in userspace cannot reliably work with namespaces, containers and other advanced Linux features."

So I wonder how the output can be different for you?
Do you not have that symlink set up?

[Christian Ehrhardt  (paelzer)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Since it seems likely to me that this is a local configuration problem, rather than a bug in Ubuntu, I'm marking this bug as Incomplete.

If indeed this is a local configuration problem, you can find pointers to get help for this sort of problem here: http://www.ubuntu.com/support/community

Or if you believe that this is really a bug, we'd be grateful if you would then provide a more complete description of the problem, explain why you believe this is a bug in Ubuntu rather than a problem specific to your system, and then change the bug status back to New.

[Andrea Ieri (aieri)]

That's right, I'm hitting this bug only because this machine is running Trusty, and mtab is not a symlink in that release.
I suppose fixing this doesn't fall within the scope of ESM support, but I'll set this to new so you can review.

[Lucas Kanashiro (lucaskanashiro)]

Thank you for taking the time to report this bug. In an effort to keep an
up-to-date and valid list of bugs to work on, I have reviewed this report
to verify it still requires effort and occurs on an Ubuntu release in
standard support, and it does not.

As you mentioned in Trusty /etc/mtab is not a symlink and that might be the reason of your issue. Apparently, this is not a security issue so it is not under the scope of the ESM support.

It is unfortunate that we were unable to resolve this defect, however
there appears to be no further action possible at this time. I am
therefore moving the bug to 'Incomplete'. If you disagree or have
new information, we would be grateful if you could please add a comment
stating why and then change the status of the bug to 'New'.

[Christian Ehrhardt  (paelzer)]

Yeah, thanks for the answers.
In that sense it is actually "Fixed" in Xenial and later.
The potential impact to SRU Trusty would be too high, and even less it would fit under the ESM support that Trusty is in nowadays.

I'll be updating bug states accordingly.