bcache makes the whole io system hang after long run time
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux-lts-xenial (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
I am using Ubuntu 14.04 (trusty) with the 4.4.x xenial kernel (the trusty kernel is way easier to make bcache crash). I have mdadm raid1 on /boot and /, backed by 2 SSDs.
I have XFS on 12 ceph directories (/var/lib/
I also have 2 unused bcache cache devices on the SSDs, without mdadm raid. This hang problem was much more frequent with the cache there, and I suspected mdadm+bcache together, so I moved it to the NVMe.
The problem happens on all these devices used as bcache: Micron S630DC-400 (firmware M013 and M017), SAMSUNG MZ7KM480HMHQ-00005 (SM863a, firmware GXM5004Q), Intel DC P3700 800GB.
If I let the machines run for a few days, and then detach and attach cache devices, it was very easy to hang it with the cache on the SSDs, but with it on the NVMe, I haven't seen that yet. The uptime on the machine was 33-34 days, and the other ones with same setup are now at 69 days 22h (that's when I changed the cache to NVMe).
For the latest hang, when the machine hangs, the text tty at the local terminal has the login prompt, but no stack trace or anything, and typing into them has no effect, not even echoing what is typed. Terminals connected previously with ssh are just hung, not responding to anything. New ssh connections fail. ping to the hung machine still replies. Soft shutdown via IPMI doesn't appear to do anything.
I will attach 2 files collected like: `ssh machine cat /dev/kmsg > cephX.kmsg` (since dmesg -w isn't supported here, and the logs do not get saved on the machine's disk since the IO system is hung).
----- Ubuntu bug reporting guidelines stuff -----
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
not including uname -a, apt-cache policy, since the kernel running now is different. It was linux-image-
Also you should likely discard similar information from the apport collect data, which is from this boot, not the previously hung one.
----- debugging procedures stuff -----
https:/
It wants a memtest, but these machines were tested in the past, and it affects more than 2 machines, so that's not useful.
I'll try to remember to try Alt+SysRq+1,t next time.
I think the other sections are about getting dmesg output, which I have already, so I'll skip that.
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-
ProcVersionSign
Uname: Linux 4.4.0-97-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.25
Architecture: amd64
Date: Tue Oct 17 10:50:03 2017
ProcEnviron:
TERM=xterm-
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: linux-lts-xenial
UpgradeStatus: No upgrade log present (probably fresh install)
Problem still exists on 4.4.0-97-generic. And in my attempt to reproduce it on a VM, I got a corrupt kernel instead, which sounds worse (could create some persistent damage, not just a hang).
Attached are the 3 scripts to reproduce the corrupt kernel, and an extra one to attach if the first failed.
- Make a VM with at least 2GB RAM.
bcache- super-show /dev/loop0 | awk '$1=="cset. uuid"{print $2}' bcache0/ bcache/ cache) bcache0/ bcache/ cache_mode
- Run crash-setup first (it uses a loop device backed by something in /dev/shm so it doesn't have to wear out an SSD).
- Then verify the cache is attached, or attach it if not.
Output of these 2 should match:
basename $(readlink /sys/block/
This should say [writeback]:
cat /sys/block/
eg. writethrough [writeback] writearound none
- Then run the setup-grep and the setup-fio at the same time, like in screen or 2 terminals. The grep is not intended to be only information output... I strongly believe that reading the files in sysfs causes crashing, as I have caused hangs and other problems many times this way in the
past.
- wait a while... could be as soon as 20 minutes, or some hours.
So far I made this VM die 3 times... it wasn't possible ot get the console output or dmesg as I wanted and I didn't save a screenshot either the first 2 times... and the 3rd time also failed, even with a serial console set up, so attached is a screenshot.