holland-mysqllvm does not remove snapshot volume after crash

[Chris Johnson]

If a server crashes while a back up is running the snapshot volume is not removed. This is particularly problematic when lvm snapshots are being using in a cluster configured for HA-LVM and the active node is fenced. Because the passive node won't start with two LV's in the VG the cluster fails to start. A work around can be put in place by adding some lines to the init.d script that will remove any snapshot volumes found but it would be nice if holland could remove it.

[Andrew Garner (muzazzi)]

I agree we should remove snapshots more aggressively. There has been a lot of talk of removing old snapshots if holland sees one exists rather than just failing due to a naming conflict (as we are currently). I wrote the code to do this last week and we should see it for 1.0.8 in the coming weeks.

I am not sure we can address this particular use case however, since holland can only remove snapshots when it actually runs or is running and has a chance to do so. If I am understanding correctly, in the case a of a crash it's unlikely holland would ever even have the chance to run before the snapshot becomes a problem. What would you like holland to do in that case?

[Chris Johnson (christopher-johnson)]

I think your right, this might be better solved by adding a small script to the RHCS configuration that looks for the snapshot volume and removes it if found then tries to start up.

However, this bring up another issue: lvm tagging. For this to work you would have to have the ability to set a tag on the snapshot volume when its created. Something like this:

# lvmcreate -L50GB --addtag @12345db1 -n mysql00_snapshop vgsan00

That will allow you to add an exception to the /etc/lvm/lvm.conf volume_list so it can mount up that LV.

[Andrew Garner (muzazzi)]

holland lvm really needs to clean up after itself when it is terminated and a stale snapshot remains. This bug has lingered too long, so let's target this for the next v1.0.10 release.

[Andrew Garner (muzazzi)]

Feature branch here: https://github.com/holland-backup/holland/tree/feature/lp_726776_lvm_snapshot_improvements

Should be included in 1.0.12

Fixed in amsterdam branch as well.