After most recent upgrade to 3.2.0-87-generic, nfs server process has extremely high I/O to /var/lib/nfs/v4recovery
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
nfs-utils (Ubuntu) |
New
|
Undecided
|
Unassigned |
Bug Description
We upgraded our 12.04 LTS on 9 July at around 18:30. Immediately after the reboot, the I/O on the / partition (sda) was extremely high. This was causing sluggish responsiveness on the NFS server who's exported directory is on a different file system (sdb) and LUN (disk).
I investigated the problem and found using iotop that the process "jbd2/sda2-8" was responsible for an extremely high number of I/O operations
Total DISK READ: 0.00 B/s | Total DISK WRITE: 13.33 M/
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
293 be/3 root 0.00 B/s 0.00 B/s 0.00 % 39.16 % [jbd2/sda2-8]
I turned on file system debugging using
echo 1 > /sys/kernel/
echo 1 >/sys/kernel/
and found that one inode was being massively addressed.
jbd2/
This inode (150) belongs to /var/lib/
Further investigation showed in dmesg that shortly after booting the directory /var/lib/
[ 99.020861] NFSD: failed to write recovery record (err -17); please check that /var/lib/
[ 99.089156] NFSD: failed to write recovery record (err -17); please check that /var/lib/
[ 99.189010] NFSD: failed to write recovery record (err -17); please check that /var/lib/
I have tried deleting and recreating the directory /var/lib/
The I/O operations per second are sometimes in excess of 1000 and typically around 500-750. This is completely different behaviour to the previous kernel where the I/O operations were in the 5-20 I/O operations per second with peaks around 30. I will attach a graph from our central EMC storage system of the LUN for a graphical view of before and after the update.
There were no changes in the parameters or shares to the NFS server and I have not been able to find any documentation about parameters that we should change to address such a problem so I can only conclude with this behaviour that this is a bug of some decsription.
Additional information about our system:
$ uname -a
Linux wsps428 3.2.0-87-generic #125-Ubuntu SMP Fri Jun 19 08:25:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ cat /proc/version_
Ubuntu 3.2.0-87.
$ lsb_release -rd
Description: Ubuntu 12.04.5 LTS
Release: 12.04
$ apt-cache policy nfs-server
nfs-server:
Installed: (none)
Candidate: (none)
Version table:
I will attach a dmesg output.
affects: | ubuntu → nfs-utils (Ubuntu) |
Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https:/ /wiki.ubuntu. com/Bugs/ FindRightPackag e. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.
To change the source package that this bug is filed about visit https:/ /bugs.launchpad .net/ubuntu/ +bug/1473948/ +editstatus and add the package name in the text box next to the word Package.
[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]