deadlock during lvm snapshot

[Tomaz Beltram]

I'm doing backup of a running mongodb using LVM snapshot. Sometimes I
run into a deadlock situation and kernel reports blocked tasks for jbd2,
mongod, dmeventd and my tar doing backup.

This happens very rarely (one in a thousand) but the effect is rather
severe as mongodb stops working. I also can't remove and unmount the
snapshot.

I have Ubuntu 16.04.1 (lvm2 2.02.133-1ubuntu10) with mongod 3.2.9 on a
64bit system.

According to the link below its a kernel race with considerably decreased chance in latest lvm2 package.

https://www.redhat.com/archives/linux-lvm/2016-November/msg00037.html

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Guillaume Penin (guillaume-penin)]

Hi all,

I think we just hit the same bug on a Production server today, with a snapshot on the root filesystem. Here are the relevant logs :

kernel: [455151.056975] INFO: task jbd2/dm-0-8:214 blocked for more than 120 seconds.
kernel: [455151.149417] INFO: task dmeventd:27193 blocked for more than 120 seconds.
kernel: [455151.241272] INFO: task cat:27522 blocked for more than 120 seconds.
...

All I/Os on the filesystem were blocked, so we had to reboot the server.

[Wolfgang Hennerbichler (wogri)]

Same here. Snapshot of "a random VM" that ran on the machine.