Volume group "cinder-volumes" not found

[Ryan Beisner]

Cinder amulet and unit tests started failing at cinder/next rev110 (which was a charm-helper sync @ http://bazaar.launchpad.net/~openstack-charmers/charms/trusty/cinder/next/revision/110).

Amulet and unit tests were passing @ cinder/next rev109 (results @ http://paste.ubuntu.com/11993736/).

NOTE: the functional test fail is with the default lvm-backed storage (not ceph-backed).

While the charm deploys and relates successfully, it appears to have a critical functional break (`cinder create` results in an error-state volume).

# Amulet
http://paste.ubuntu.com/11977164/
...
2015-07-31 17:38:08,423 resource_reaches_status DEBUG: Volume status wait: expected, actual status = error, available
2015-07-31 17:38:08,423 resource_reaches_status DEBUG: 090de824-3cb8-474f-8c78-fc92a2dd3216 never reached expected status: available
Cinder volume failed to reach expected state.
...

# Unit
http://paste.ubuntu.com/11977142/
...
Expected: [call('/dev/vdb'), call('/dev/vdc')]
Actual: [call('/dev/vdc')]
...

# cinder-volume.log
http://paste.ubuntu.com/11977176/
...
2015-07-31 16:51:04.006 15304 TRACE cinder.volume.manager Stderr: u' Volume group "cinder-volumes" not found\n'
...
2015-07-31 16:50:47.414 13853 TRACE cinder.openstack.common.threadgroup OperationalError: (OperationalError) (1054, "Unknown column 'services.disabled_reason' in 'field list'") 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason \nFROM services \nWHERE services.deleted = false AND services.host = %s AND services.`binary` = %s \n LIMIT %s' ('juju-osci-sv05-machine-1', 'cinder-volume', 1)

[Jason Hobbs (jason-hobbs)]

I'm not getting the unit test failure on r112, but that's on vivid.

[Jason Hobbs (jason-hobbs)]

I do get the unit test failures on trusty.

[Ryan Beisner (1chb1n)]

Observation on a manual deploy, seeing ^ the same as reported above, also:

root@juju-beis0-machine-5:/var/log/cinder# sudo vgs
  No volume groups found
root@juju-beis0-machine-5:/var/log/cinder# sudo lvs
  No volume groups found

[Ryan Beisner (1chb1n)]

NOTE: this is with lvm-backed storage (not ceph-backed).

[Ryan Beisner (1chb1n)]

Existing logic in the cinder charm causes it to exclude disks which are already mounted, via a check to charm-helper:is_device_mounted.

A recent charm-helpers fix resolved a bug where it was previously returning False even when a device was mounted (http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/revision/409).

With that c-h fix, the cinder charm started getting an accurate device list for available (not-mounted) disk devices.

In the test environment, the 2nd disk (ephemeral) is already mounted @ /mnt, which is why cinder finds no devices available to use.

[Ryan Beisner (1chb1n)]

With the new ephemeral-unmount option added to the cinder charm, the following amulet tests will need to be updated to include the new cinder config data before their tests will pass:

ceph
ceph-osd
ceph-radosgw
cinder-ceph
keystone
rabbitmq-server

[Ryan Beisner (1chb1n)]

Correction. Only non-ceph-backed cinder test cases should be affected. Of that ^ list, only the following should need an amulet test update:
keystone