A severe performance degradation while syncing md raid after the upgrade to 15.10

[Pavol Domin]

I run Ubuntu from the fakerade (imsm) raid5 volume. If the volume is syncing, the system response is severely degraded. There is no problem if I boot the same system from the previous kernel 3.19.0-32. Some numbers:

[md125_resync] kernel thread CPU consumption: 94 per-mill compared to 10 in 3.19 kernel
[md125_raid5] kernel thread CPU consumption: 243 per-mill compared to 86 in 3.19 kernel
1MB direct read from md array: 0.5s compared to 0.01s in 3.19 kernel
1MB direct read from md raid member: 2.2s compared to 0.02s in 3.19 kernel

As a result, it's practically impossible to work in 4.2 kernel if md raid is syncing (even the boot takes several minutes here), whereas in 3.19 I don't feel any performance degradation at all, regardless the raid volume is syncing or not.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-4.2.0-22-generic 4.2.0-22.27
ProcVersionSignature: Ubuntu 4.2.0-22.27-generic 4.2.6
Uname: Linux 4.2.0-22-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: palo 3869 F.... pulseaudio
 /dev/snd/pcmC0D0p: palo 3869 F...m pulseaudio
 /dev/snd/controlC0: palo 3869 F.... pulseaudio
CurrentDesktop: Unity
Date: Thu Jan 7 18:51:39 2016
HibernationDevice: RESUME=UUID=f9900a93-c16d-4cae-9098-194fdf3abbfe
InstallationDate: Installed on 2011-09-16 (1574 days ago)
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110427.1)
MachineType: System manufacturer System Product Name
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.2.0-22-generic root=/dev/mapper/vg00-lv_ubuntu_root ro init=/sbin/upstart
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-22-generic N/A
 linux-backports-modules-4.2.0-22-generic N/A
 linux-firmware 1.149.3
RfKill:

SourcePackage: linux
UpgradeStatus: Upgraded to wily on 2016-01-02 (4 days ago)
dmi.bios.date: 02/23/2009
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2102
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: P5Q-PRO
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2102:bd02/23/2009:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP5Q-PRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.4 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4-rc8-wily

[Joseph Salisbury (jsalisbury)]

If the mainline kernel does not exhibit the bug, we can perform a reverse bisect to find the fix. If the bug does still exist in mainline, we can perform a regular bisect to identify the commit that introduced this regression.

[Pavol Domin (pavol-domin)]

Tested with the mainline kernel: the bug is reproducible also in 4.4.0-040400rc8-generic.

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

Once this bug is reported upstream, please add the tag: 'kernel-bug-reported-upstream'.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Joseph Salisbury (jsalisbury)]

If we don't receive any feedback from upstream, we can perform a kernel bisect to identify the commit that introduced the regression.

[penalvch (penalvch)]

Pavol Domin, in order to maximize the probability of upstream addressing this issue as fast as possible, you would want to fully commit bisect from kernel 3.19 to 4.2 in order to identify the last good kernel commit, followed immediately by the first bad one. Could you please do this at your convenience following https://wiki.ubuntu.com/Kernel/KernelBisection ?

Please note, finding adjacent kernel versions is not fully commit bisecting.

Thank you for your understanding.