| 2012-05-30 15:50:00 |
perpetualrabbit |
description |
Problem:
--------------------------------------------------------------
I just had to remove ubuntu server 12.04 to install redhat enterprise linux 6. The intermittent slowness was
completely unacceptable for the users, who have workstations with /home mounted with nfs4 on this server.
The mail server, also accesses the /home because the /home/$USER/Maildir directories are there.
Using nfs4, the kernel nfs threads caused enormous load.
The users had frozen desktops (greyed out windows) and mail slowed or arrived days later as a result.
With RHEL6, all nfs4 problems are completely gone. I used the exact same /etc/exports file, and the same
settings and mount options on the workstations, the same number of nfs threads.
Both the redhat and ubuntu systems are KVM virtual guests on an redhat 6 virtual host (one of 3 actually).
The storage backend is a very fast equallogic array, which exports iscsi targets to the virtual hosts.
I am sorry, but I have to conclude the current nfs4 implementation of ubuntu server 12.04 is NOT fit for use.
A complete university department suffered for weeks while I tried to solve the problems with ubuntu, but
in the end it was decided to install redhat instead, re-using the same iscsi targets for system, home and data.
A missed chance for ubuntu...
Therefore I urge Canonical's people to classify this bug as critical.
Also I think quality assurance should have caught this bug before shipping.
Analysis
------------------------------------------------------------
LOAD:
The nfs threads cause the kernel to use enormous amounts of 'sy' time as measured in top.
I will attach a sample of top's output, of a particular _quiet_ time on the network. Load is 7.82.
On busier moments, the load went through the roof, beyond 50 and further. It consumes
actual CPU cycles.
Each thread consumes upto 30% of a cpu core. I enabled 128 threads.
rx an tx block sizes are 32768 on the clients. Both server and clients used async, both on redhat and ubuntu.
SYSTEM vs IO-WAIT:
The replacement redhat system can surely be overloaded, but then it does not consume CPU cycles doing so. Top does report high load, but it spends in in the 'wa' state. This indicates it is simply waiting for its backend iscsi devices to complete writes. I tested this by simultaneously letting all workstations write multi-gigabyte files with dd to /home.
On ubuntu, the nfs threads spend their time in 'sy', doing who-knows-what.
LOGS:
Nothing at all appears in the logs. But when I set bitwise debug options in the /proc/sys/sunrpc/*debug files,
lots of log entries appear. Those seem like normal NFS protocol messages to me though.
I also tried to discover what was happening with wireshark, but the traffic looks like normal nfs4 traffic to me.
SLOWNESS:
That is the thing. The ubuntu nfs server is actually NOT slow at all. During my dd tests, it wrote half a gigabyte per second to its iscsi backends. It's _throughput_ is better than that of the redhat server.
As far as I can tell, it falls down because it makes client side processes that want to do IO wait on other writes. A simple 'ls' has to wait until a write has been completed. And both server and client used async nfs. People's firefoxes freeze all the time because firefox need to read and write a lot to its cache and other files in the .mozilla directory.
The dovecot imap server almost grinds to a halt trying to write all those little files in people's /home/$USER/Maildir's.
The problems go on and on. Basically, a complete network of workstations is almost unusable because of this.
Upfront tests were done of course, but showed only the excellent throughput but not the appalling `waiting´ behaviour.
With redhat 6 there is no such problem.
SITUATION:
People use their own, and each others linux workstations for science, doing large calculations and writing a lot of big and small files to nfs. The nfs server serves /export/home, and also raw data storage from /export/data with nfs4. The clients mount those under /home and /data/misc respectively. Also there is a read-only software mount for certain scientific packages.
CONFIG:
client fstab lines:
#### nfs entries ###
sw.lorentz.leidenuniv.nl:/sw /sw nfs4 hard,intr,ro,tcp,rsize=32768,wsize=32768,bg,acl,async
home.lorentz.leidenuniv.nl:/home /home nfs4 hard,intr,rw,tcp,rsize=32768,wsize=32768,bg,acl,async
server exports file:
/export 132.229.227.0/24(ro,sync,insecure,root_squash,no_subtree_check,nohide,fsid=0)\
132.229.216.128/26(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)\
132.229.226.3(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)\
132.229.226.4(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)
/export/home 132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.214.41(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
/export/data 132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
/export/sw 132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
root@gaia:~# lsb_release -rd
Description: Ubuntu 12.04 LTS
Release: 12.04
root@gaia2:~# dpkg -l | grep -E 'nfs|linux-image'
ii libnfsidmap2 0.25-1ubuntu2 NFS idmapping library
ii linux-image-3.2.0-18-generic 3.2.0-18.29 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-19-generic 3.2.0-19.31 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-20-generic 3.2.0-20.33 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-21-generic 3.2.0-21.34 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-23-generic 3.2.0-23.36 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-server 3.2.0.24.26 Linux kernel image on Server Equipment.
ii nfs-common 1:1.2.5-3ubuntu3 NFS support files common to client and server
ii nfs-kernel-server 1:1.2.5-3ubuntu3 support for NFS kernel server
ii nfswatch 4.99.11-1 Program to monitor NFS traffic for the console
WHAT I EXPECTED TO HAPPEN
----------------------------------
A fast and responsive nfs service.
WHAT HAPPENED INSTEAD
-----------------------------
I got fast, but also intermittently totally unresponsive. |
Problem:
--------------------------------------------------------------
I just had to remove ubuntu server 12.04 to install redhat enterprise linux 6. The intermittent slowness was
completely unacceptable for the users, who have workstations with /home mounted with nfs4 on this server.
The mail server, also accesses the /home because the /home/$USER/Maildir directories are there.
Using nfs4, the kernel nfs threads caused enormous load.
The users had frozen desktops (greyed out windows) and mail slowed or arrived days later as a result.
With RHEL6, all nfs4 problems are completely gone. I used the exact same /etc/exports file, and the same
settings and mount options on the workstations, the same number of nfs threads.
Both the redhat and ubuntu systems are KVM virtual guests on an redhat 6 virtual host (one of 3 actually).
The storage backend is a very fast equallogic array, which exports iscsi targets to the virtual hosts.
I am sorry, but I have to conclude the current nfs4 implementation of ubuntu server 12.04 is NOT fit for use.
A complete university department suffered for weeks while I tried to solve the problems with ubuntu, but
in the end it was decided to install redhat instead, re-using the same iscsi targets for system, home and data.
A missed chance for ubuntu...
Therefore I urge Canonical's people to classify this bug as critical.
Also I think quality assurance should have caught this bug before shipping.
Analysis
------------------------------------------------------------
LOAD:
The nfs threads cause the kernel to use enormous amounts of 'sy' time as measured in top.
I will attach a sample of top's output, of a particular _quiet_ time on the network. Load is 7.82.
On busier moments, the load went through the roof, beyond 50 and further. It consumes
actual CPU cycles.
Each thread consumes upto 30% of a cpu core. I enabled 128 threads.
rx an tx block sizes are 32768 on the clients. Both server and clients used async, both on redhat and ubuntu.
SYSTEM vs IO-WAIT:
The replacement redhat system can surely be overloaded, but then it does not consume CPU cycles doing so. Top does report high load, but it spends in in the 'wa' state. This indicates it is simply waiting for its backend iscsi devices to complete writes. I tested this by simultaneously letting all workstations write multi-gigabyte files with dd to /home.
On ubuntu, the nfs threads spend their time in 'sy', doing who-knows-what.
LOGS:
Nothing at all appears in the logs. But when I set bitwise debug options in the /proc/sys/sunrpc/*debug files,
lots of log entries appear. Those seem like normal NFS protocol messages to me though.
I also tried to discover what was happening with wireshark, but the traffic looks like normal nfs4 traffic to me.
SLOWNESS:
That is the thing. The ubuntu nfs server is actually NOT slow at all. During my dd tests, it wrote half a gigabyte per second to its iscsi backends. It's _throughput_ is better than that of the redhat server.
As far as I can tell, it falls down because it makes client side processes that want to do IO wait on other writes. A simple 'ls' has to wait until a write has been completed. And both server and client used async nfs. People's firefoxes freeze all the time because firefox need to read and write a lot to its cache and other files in the .mozilla directory.
The dovecot imap server almost grinds to a halt trying to write all those little files in people's /home/$USER/Maildir's.
The problems go on and on. Basically, a complete network of workstations is almost unusable because of this.
Upfront tests were done of course, but showed only the excellent throughput but not the appalling `waiting´ behaviour.
With redhat 6 there is no such problem.
SITUATION:
People use their own, and each others linux workstations for science, doing large calculations and writing a lot of big and small files to nfs. The nfs server serves /export/home, and also raw data storage from /export/data with nfs4. The clients mount those under /home and /data/misc respectively. Also there is a read-only software mount for certain scientific packages.
CONFIG:
client fstab lines:
#### nfs entries ###
sw.lorentz.leidenuniv.nl:/sw /sw nfs4 hard,intr,ro,tcp,rsize=32768,wsize=32768,bg,acl,async
home.lorentz.leidenuniv.nl:/home /home nfs4 hard,intr,rw,tcp,rsize=32768,wsize=32768,bg,acl,async
server exports file:
/export 132.229.227.0/24(ro,sync,insecure,root_squash,no_subtree_check,nohide,fsid=0)\
132.229.216.128/26(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)\
132.229.226.3(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)\
132.229.226.4(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)
/export/home 132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.214.41(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
/export/data 132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
/export/sw 132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
root@gaia:~# lsb_release -rd
Description: Ubuntu 12.04 LTS
Release: 12.04
root@gaia2:~# dpkg -l | grep -E 'nfs|linux-image'
ii libnfsidmap2 0.25-1ubuntu2 NFS idmapping library
ii linux-image-3.2.0-18-generic 3.2.0-18.29 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-19-generic 3.2.0-19.31 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-20-generic 3.2.0-20.33 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-21-generic 3.2.0-21.34 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-23-generic 3.2.0-23.36 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-server 3.2.0.24.26 Linux kernel image on Server Equipment.
ii nfs-common 1:1.2.5-3ubuntu3 NFS support files common to client and server
ii nfs-kernel-server 1:1.2.5-3ubuntu3 support for NFS kernel server
ii nfswatch 4.99.11-1 Program to monitor NFS traffic for the console
WHAT I EXPECTED TO HAPPEN
----------------------------------
A fast and responsive nfs service.
WHAT HAPPENED INSTEAD
-----------------------------
I got fast, but also intermittently totally unresponsive.
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info: Error: [Errno 2] No such file or directory
Card0.Amixer.values: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 17.280028] eth0: no IPv6 routers present
DistroRelease: Ubuntu 12.04
HibernationDevice: RESUME=UUID=aafa9be4-19ac-4d74-a853-a5532ddedf5d
InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Alpha amd64 (20120313)
IwConfig:
lo no wireless extensions.
eth0 no wireless extensions.
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Red Hat KVM
NonfreeKernelModules: nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc ext2 snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd psmouse serio_raw lp parport i2c_piix4 soundcore snd_page_alloc virtio_balloon mac_hid floppy
Package: linux (not installed)
ProcEnviron:
LANGUAGE=en_US:en
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-24-generic root=/dev/mapper/gaia-root ro
ProcVersionSignature: Ubuntu 3.2.0-24.39-generic 3.2.16
RelatedPackageVersions:
linux-restricted-modules-3.2.0-24-generic N/A
linux-backports-modules-3.2.0-24-generic N/A
linux-firmware 1.79
RfKill: Error: [Errno 2] No such file or directory
Tags: precise
Uname: Linux 3.2.0-24-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Seabios
dmi.bios.version: 0.5.1
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.modalias: dmi:bvnSeabios:bvr0.5.1:bd01/01/2007:svnRedHat:pnKVM:pvrRHEL6.2.0PC:cvnRedHat:ct1:cvr:
dmi.product.name: KVM
dmi.product.version: RHEL 6.2.0 PC
dmi.sys.vendor: Red Hat |
|
