bcache ioutil to 100% needs node reboot (no kern logs related)

Bug #1662573 reported by Alvaro Uria
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

bcache (bcache0 in this sample) ioutil goes to 100% and avgqu-sz also gets affected. Bcache slaves don't get affected, but does affect storage performance.

See excerpt (bcache0):
http://pastebin.ubuntu.com/23948463/

* Bcache blocks are running Ceph OSDs.
* HW is PowerEdge R730xd
* Kernel version is: 4.4.0-51-generic
* Ubuntu 14.04.5 LTS
* bcache-tools 1.0.7-1~14.04.1

Please let me know if you would need further detail.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Feb 7 12:45 seq
 crw-rw---- 1 root audio 116, 33 Feb 7 12:45 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.21
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 14.04
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 002 Device 002: ID 8087:8002 Intel Corp.
 Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub
 Bus 001 Device 002: ID 8087:800a Intel Corp.
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Dell Inc. PowerEdge R730xd
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-51-generic root=UUID=dc9574d7-cd8b-4db0-af85-db73ea17df65 ro console=tty0 console=ttyS0,115200n8
ProcVersionSignature: Ubuntu 4.4.0-51.72~14.04.1-generic 4.4.30
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-51-generic N/A
 linux-backports-modules-4.4.0-51-generic N/A
 linux-firmware 1.127.22
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty uec-images
Uname: Linux 4.4.0-51-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 08/28/2014
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.0.4
dmi.board.name: 0H21J3
dmi.board.vendor: Dell Inc.
dmi.board.version: A04
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.0.4:bd08/28/2014:svnDellInc.:pnPowerEdgeR730xd:pvr:rvnDellInc.:rn0H21J3:rvrA04:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R730xd
dmi.sys.vendor: Dell Inc.

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1662573

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.10 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc7

Changed in linux (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Alvaro Uria (aluria) wrote : BootDmesg.txt

apport information

tags: added: apport-collected trusty uec-images
description: updated
Revision history for this message
Alvaro Uria (aluria) wrote : CRDA.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : Lspci.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : NonfreeKernelModules.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : ProcModules.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : UdevDb.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote : UdevLog.txt

apport information

Revision history for this message
Alvaro Uria (aluria) wrote :

apport-collect with edited hostname

Revision history for this message
Alvaro Uria (aluria) wrote :

In reply to comment #2, we're using ksplice to patch the kernel, but we haven't upgraded the ubuntu package, lately.

I'm afraid but this is a production node and I can't use it for testing. In case it helps, this node hadn't had this issue before (62 days running) and after yesterday's reboot, it has been behaving as expected (~0% ioutil on bcache0).

Changed in linux (Ubuntu):
status: Incomplete → New
Revision history for this message
Brad Figg (brad-figg) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.