Can't remove VGs when partial LVs exist

[Jason Hobbs]

I'm seeing this error:

2015-06-25 04:29:36 DEBUG worker.uniter.jujuc server.go:139 hook context id "cinder/0-config-changed-3607276790042541683"; dir "/var/lib/juju/agents/unit-cinder-0/charm"
2015-06-25 04:29:37 INFO config-changed Couldn't find device with uuid qrTOA0-RXw2-iTx5-iMnC-ZT9L-e6Nk-rAc1Y1.
2015-06-25 04:29:37 INFO config-changed Couldn't find device with uuid qrTOA0-RXw2-iTx5-iMnC-ZT9L-e6Nk-rAc1Y1.
2015-06-25 04:29:37 INFO config-changed Couldn't find device with uuid qrTOA0-RXw2-iTx5-iMnC-ZT9L-e6Nk-rAc1Y1.
2015-06-25 04:29:37 INFO config-changed Couldn't find device with uuid qrTOA0-RXw2-iTx5-iMnC-ZT9L-e6Nk-rAc1Y1.
2015-06-25 04:29:37 INFO config-changed Couldn't find device with uuid qrTOA0-RXw2-iTx5-iMnC-ZT9L-e6Nk-rAc1Y1.
2015-06-25 04:29:37 INFO config-changed No physical volume label read from /dev/loop0
2015-06-25 04:29:37 INFO config-changed Failed to read physical volume "/dev/loop0"
2015-06-25 04:29:37 INFO config-changed No physical volume label read from /dev/loop0
2015-06-25 04:29:37 INFO config-changed Failed to read physical volume "/dev/loop0"
2015-06-25 04:29:53 INFO config-changed Creating new GPT entries.
2015-06-25 04:29:53 INFO config-changed Warning: The kernel is still using the old partition table.
2015-06-25 04:29:53 INFO config-changed The new table will be used at the next reboot.
2015-06-25 04:29:53 INFO config-changed GPT data structures destroyed! You may now partition the disk using fdisk or
2015-06-25 04:29:53 INFO config-changed other utilities.
2015-06-25 04:29:53 INFO config-changed Warning: The kernel is still using the old partition table.
2015-06-25 04:29:53 INFO config-changed The new table will be used at the next reboot.
2015-06-25 04:29:53 INFO config-changed The operation has completed successfully.
2015-06-25 04:29:54 INFO config-changed 1+0 records in
2015-06-25 04:29:54 INFO config-changed 1+0 records out
2015-06-25 04:29:54 INFO config-changed 1048576 bytes (1.0 MB) copied, 0.00360436 s, 291 MB/s
2015-06-25 04:29:54 INFO config-changed 100+0 records in
2015-06-25 04:29:54 INFO config-changed 100+0 records out
2015-06-25 04:29:54 INFO config-changed 51200 bytes (51 kB) copied, 0.000633994 s, 80.8 MB/s
2015-06-25 04:29:54 INFO config-changed Physical volume "/dev/loop0" successfully created
2015-06-25 04:29:54 INFO config-changed Couldn't find device with uuid qrTOA0-RXw2-iTx5-iMnC-ZT9L-e6Nk-rAc1Y1.
2015-06-25 04:29:54 INFO config-changed WARNING: Partial LV volume-f9770314-d037-4176-b2ad-407985063579 needs to be repaired or removed.
2015-06-25 04:29:54 INFO config-changed WARNING: There are still partial LVs in VG cinder-volumes.
2015-06-25 04:29:54 INFO config-changed To remove them unconditionally use: vgreduce --removemissing --force.
2015-06-25 04:29:54 INFO config-changed Proceeding to remove empty missing PVs.
2015-06-25 04:29:54 INFO config-changed Traceback (most recent call last):
2015-06-25 04:29:54 INFO config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 519, in <module>
2015-06-25 04:29:54 INFO config-changed hooks.execute(sys.argv)
2015-06-25 04:29:54 INFO config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 557, in execute
2015-06-25 04:29:54 INFO config-changed self._hooks[hook_name]()
2015-06-25 04:29:54 INFO config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/host.py", line 312, in wrapped_f
2015-06-25 04:29:54 INFO config-changed f(*args, **kwargs)
2015-06-25 04:29:54 INFO config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 131, in config_changed
2015-06-25 04:29:54 INFO config-changed conf['remove-missing'])
2015-06-25 04:29:54 INFO config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 377, in configure_lvm_storage
2015-06-25 04:29:54 INFO config-changed reduce_lvm_volume_group_missing(volume_group)
2015-06-25 04:29:54 INFO config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 313, in reduce_lvm_volume_group_missing
2015-06-25 04:29:54 INFO config-changed subprocess.check_call(['vgreduce', '--removemissing', volume_group])
2015-06-25 04:29:54 INFO config-changed File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
2015-06-25 04:29:54 INFO config-changed raise CalledProcessError(retcode, cmd)
2015-06-25 04:29:54 INFO config-changed subprocess.CalledProcessError: Command '['vgreduce', '--removemissing', u'cinder-volumes']' returned non-zero exit status 5
2015-06-25 04:29:54 INFO juju.worker.uniter.context context.go:534 handling reboot
2015-06-25 04:29:54 ERROR juju.worker.uniter.operation runhook.go:107 hook "config-changed" failed: exit status 1

[Greg Lutostanski (lutostag)]

Not sure if the following scenarios should be mentioned in this bug or new bugs should be created (both similar but unique tracebacks included below in respective order):

1. A volume with that name already exists (perhaps on a different disk)
2. A volume group is assumed to exist when it does not

config-changed Physical volume "/dev/loop0" successfully created
config-changed A volume group called cinder-volumes already exists.
config-changed Traceback (most recent call last):
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 519, in <module>
config-changed hooks.execute(sys.argv)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 557, in execute
config-changed self._hooks[hook_name]()
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/host.py", line 312, in wrapped_f
config-changed f(*args, **kwargs)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 131, in config_changed
config-changed conf['remove-missing'])
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 372, in configure_lvm_storage
config-changed create_lvm_volume_group(volume_group, new_devices[0])
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/storage/linux/lvm.py", line 105, in create_lvm_volume_group
config-changed check_call(['vgcreate', volume_group, block_device])
config-changed File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
config-changed raise CalledProcessError(retcode, cmd)
config-changed subprocess.CalledProcessError: Command '['vgcreate', u'cinder-volumes', '/dev/loop0']' returned non-zero exit status 5

config-changed Volume group "cinder-volumes" not found
config-changed Traceback (most recent call last):
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 519, in <module>
config-changed hooks.execute(sys.argv)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 557, in execute
config-changed self._hooks[hook_name]()
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/host.py", line 312, in wrapped_f
config-changed f(*args, **kwargs)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 131, in config_changed
config-changed conf['remove-missing'])
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 377, in configure_lvm_storage
config-changed reduce_lvm_volume_group_missing(volume_group)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 313, in reduce_lvm_volume_group_missing
config-changed subprocess.check_call(['vgreduce', '--removemissing', volume_group])
config-changed File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
config-changed raise CalledProcessError(retcode, cmd)
config-changed subprocess.CalledProcessError: Command '['vgreduce', '--removemissing', u'cin...

[Jason Hobbs (jason-hobbs)]

I'm testing lp:~jason-hobbs/charms/trusty/cinder/force-vgreduce to see if it resolves this issue.

[Corey Bryant (corey.bryant)]

Jason, If your patch works I think we should put that under a different config option, perhaps 'remove-missing-force' to keep behavior of 'remove-missing' they same.

[Corey Bryant (corey.bryant)]

As for the failure of 'vgcreate cinder-volumes /dev/loop0' when a volume with that name already exists. I'm thinking we might be able to use --force there too, if the 'overwrite' option is true.

[Jason Hobbs (jason-hobbs)]

With this fix we stopped seeing the issue I reported, but are still seeing at least one of the issues Greg mentioned, so I dug into this a bit more to see what's happening in these different cases.

The prompt to use "--force" comes up when attempting to use 'vgreduce --removemissing cinder-volumes'.

To recreate this you error you need at least 2 devices. Create a volume group named 'cinder-volumes' on the devices, then create a big enough logical volume (or more than one) to use physical extents from both devices. Then use 'dd' to overwrite one of the devices. Then run 'vgreduce --removemissing cinder-volumes'. This will produce an error saying there are still partial LVs in VG cinder-volumes, since cinder-volumes extended across both devices and one is now missing. It doesn't want to remove the missing PV from cinder-volumes because there are partial LVs, so it's asking for confirmation.

We could hit this scenario if system with multiple storage devices has an existing PV on one of the devices configured for use by cinder, and that PV is part of a VG with other PVs now missing (overwritten and used for other things since the node was last used for cinder, or maybe was just a loopback device on filesystem that's gone now).

Using --force will cause vgreduce to remove the missing PV, and to remove any LVs that used it. The charm will function then. The ability to use '--force' could be added as an option to the charm.

I'll submit a patch with a new 'remove-missing-force' option as you suggest Corey.

Greg mentioned two other failures - I think those are both unique bugs and separate from this one. I filed them as bug 1476418 and bug 1476419.