lvm snapshot is not being autoextended

[Doug Fultz]

After setting auto extend threshold and rebooting, lvm snapshots are not being auto extended.

Steps to recreate:
1. Using Ubuntu 14.04 LTS
2. Set snapshot_autoextend_threshold to less than 100
# grep autoext /etc/lvm/lvm.conf
    snapshot_autoextend_threshold = 50
    snapshot_autoextend_percent = 20
3. Reboot server to have lvm reread lvm.conf
4. Create a snapshot of a mounted logical volume
# lvcreate -s /dev/testVG/testLV -L 1G -n testLVsnap
5. Make modifications to mounted logical volume
6. List logical volumes and snapshots
# lvs
  LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
  testLV testVG owi-aos-- 10.00g
  testLVsnap testVG swi-a-s-- 1.00g testLV 95.04

What I expected to have happened:
Once Data% of snapshot reached the snapshot_autoextend_threshold, the snapshot logical volume should have been auto extended.

[Noam Helfman (nhelfman)]

I have seen a similar issue with Ubuntu 14.10

Repro steps:

root@ubuntu-amd64:~# lsb_release -cd
Description: Ubuntu 14.10
Codename: utopic

# after reboot
root@ubuntu-amd64:~# cat /etc/lvm/lvm.conf | grep " snapshot_autoextend"
    snapshot_autoextend_threshold = 80
    snapshot_autoextend_percent = 20

root@ubuntu-amd64:~# vgs
  VG #PV #LV #SN Attr VSize VFree
  vg1 1 0 0 wz--n- 20.00g 20.00g

root@ubuntu-amd64:~# lvcreate -n vol1 -L 100MB vg1
  Logical volume "vol1" created
root@ubuntu-amd64:~# lvs
  LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
  vol1 vg1 -wi-a---- 100.00m

root@ubuntu-amd64:~# lvs
  LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
  snap1 vg1 swi-a-s-- 12.00m vol1 0.00
  vol1 vg1 owi-a-s-- 100.00m

root@ubuntu-amd64:~# dd if=/dev/zero of=/dev/vg1/vol1 count=11 bs=1MB
11+0 records in
11+0 records out
11000000 bytes (11 MB) copied, 0.087546 s, 126 MB/s
root@ubuntu-amd64:~# lvs
  LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
  snap1 vg1 swi-a-s-- 12.00m vol1 87.83
  vol1 vg1 owi-a-s-- 100.00m

At this point I expected the snap1 volume to be auto extended based on the 80% setting in lvm.conf. However as can be seen the snapshot is 87% and the size stays as 12MB. If I continue writing the snapshot gets fill and disabled.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Donk (donk)]

To solve this issue, you need to install the package "dmeventd" and, in the file "/etc/lvm/lvm.conf", set the value "monitoring" to 1.

[Josip Rodin (joy)]

On Ubuntu 14, it doesn't seem to work with monitoring = 1 set, either. I had also tried doing vgchange --monitor y --poll y volumegroup, and lvcreate --snapshot --monitor y ... and there was no effect.

Testing on Ubuntu 16 now

[Josip Rodin (joy)]

It looks like it works on Ubuntu 16, syslog contains things like:

Feb 10 11:28:55 hostname lvm[1704]: WARNING: Snapshot vgname-lvname is now 80.03% full.
Feb 10 11:28:55 hostname lvm[1704]: Size of logical volume vgname/lvname changed from 1.00 GiB (256 extents) to 1.20 GiB (308 extents).
Feb 10 11:28:55 hostname lvm[1704]: Logical volume lvname successfully resized.

Feb 10 11:29:12 hostname lvm[1704]: WARNING: Snapshot vgname-lvname is now 85.65% full.
Feb 10 11:29:12 hostname lvm[1704]: Size of logical volume vgname/lvname changed from 1.80 GiB (462 extents) to 2.17 GiB (555 extents).
Feb 10 11:29:12 hostname lvm[1704]: Logical volume lvname successfully resized.

[Donk (donk)]

@Josip Rodin:
On Ubuntu 14, have you installed the package "dmeventd"?

[Josip Rodin (joy)]

@Donk yes, of course.

% ps auxfw | grep '[d]'meventd
root 3325 0.1 0.0 574972 18956 ? S<Lsl Jan30 25:34 /sbin/dmeventd

% dpkg -l dmeventd | grep ^.i
ii dmeventd 2:1.02.77-6ubuntu2 amd64 Linux Kernel Device Mapper event daemon

% egrep '^[^#]*(snapshot_autoextend|monitoring)' /etc/lvm/lvm.conf
    snapshot_autoextend_threshold = 80
    snapshot_autoextend_percent = 20
    monitoring = 1

And syslog actually mentions:

Feb 12 06:18:04 hostname lvm[3325]: Monitoring snapshot vgname-lvname

Yet, it never notices the increases, for example:

Feb 12 06:18:07 hostname kernel: [1102775.345351] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Feb 12 06:18:07 drop kernel: [1102775.345746] Buffer I/O error on device dm-0, logical block 60118
...

% sudo dmsetup info --noheadings -c -o name /dev/dm-0
vgname-lvname

Now you could say this particular one happened three seconds later and it never had the chance, but it actually just never happens, over months of usage.

[Josip Rodin (joy)]

I did some more testing with that trusty system and it does appear to be working much better than before (i.e. not at all), though it seems there are obfuscating circumstances - AFAICT it looks like it recognizes lost causes (100.00% full snapshots) that it then no longer tries to extend, but doesn't actually inform about them in a very meaningful way.

For example:

Feb 12 12:39:57 hostname lvm[3325]: Logical volume lvname successfully resized
Feb 12 12:39:59 hostname lvm[3325]: No longer monitoring snapshot vgname-lvname
Feb 12 12:40:00 hostname lvm[3325]: dm_task_run failed, errno = 22, Invalid argument

Here it did the extend but the next iteration didn't win the race and just silently dropped the ball afterwards...?

Feb 12 12:44:39 hostname lvm[3325]: Snapshot vgname-lvname is now 92% full.
Feb 12 12:44:40 hostname lvm[3325]: device-mapper: waitevent ioctl on failed: Interrupted system call
Feb 12 12:44:40 hostname lvm[3325]: device-mapper: waitevent ioctl on failed: Interrupted system call
Feb 12 12:44:44 hostname lvm[3325]: Extending logical volume lvname to 252.00 MiB
Feb 12 12:44:44 hostname lvm[3325]: Monitoring snapshot vgname-lvname
Feb 12 12:44:44 hostname lvm[3325]: Logical volume lvname successfully resized
Feb 12 12:44:46 hostname lvm[3325]: No longer monitoring snapshot vgname-lvname

Here it looked like it was winning the race but inbetween it noticing it should extend and actually extending, it actually lost the race, and then the last message came with a larger lag...?

So it does appear that the original issue is fixed with the latest trusty packages, with non-trivial caveats.