Write performance regression severely affecting hpsa controllers
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Upgrading our HP(E) servers to xenial we have discovered a severe performance regression affecting write performance on Smart Array RAID logical drives.
Firmware is up to date with latest HPE SPP release at the time of testing.
This performance regression has been verified on multiple sites with different HPE systems and OS:s, downgrading kernel brings back the expected performance.
We are NOT seeing this on Dell hardware with H730P (LSI based) controllers, using the same OS installs.
Our test system setup:
Ubuntu 16.04.2 LTS
Server: HP(E) DL380e, 36 G RAM, 1x E5-2420 CPU
RAID Controller: Smart Array P420 2GB FBWC
RAID setup: 24x 500G SAS HDDs in RAID50 with 2 parity groups.
File system: xfs
We are also seeing this issue on setups using DL380e with P430 controller and 14 HDDs in RAID6, but those systems are not available for testing.
Fast/normal (previous) bulk IO performance is approx 1600-1800 MB/s sustained read and write using a simple dd bs=256k based test.
Slow/performance regression reduces the write performance to approx 500-600 MB/s sustained using the same tests and filesystem.
We have tested using the following OS installs and kernels:
Precise:
All tested Ubuntu kernels are fast.
Trusty:
trusty 3.13.0.110.118 fast
utopic 3.16.0.77.68 fast
vivid 3.19.0.80.62 slow
xenial 4.4.0.64.50 slow
Xenial:
xenial 4.4.0.64.68 slow
hwe 4.8.0.39.10 slow
mainline 3.12.64-
mainline 3.16.41-
mainline 3.17.8-
mainline 3.18.0-
mainline 3.18.12-
mainline 3.18.18-
mainline 3.18.21-
mainline 3.18.22-
mainline 3.18.23-
mainline 3.18.24-
mainline 3.18.47-
mainline 4.10.1-
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Feb 28 10:29 seq
crw-rw---- 1 root audio 116, 33 Feb 28 10:29 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
IwConfig: Error: [Errno 2] No such file or directory
MachineType: HP ProLiant DL380e Gen8
Package: linux (not installed)
PciMultimedia:
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 1.157.8
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial xenial
Uname: Linux 4.4.0-64-generic x86_64
UnreportableReason: The report belongs to a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: False
dmi.bios.date: 08/02/2014
dmi.bios.vendor: HP
dmi.bios.version: P73
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:
dmi.product.name: ProLiant DL380e Gen8
dmi.sys.vendor: HP
This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:
apport-collect 1668557
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.