drop ceph support from cinder charm

[Barry Price]

Creating instances and volumes seems to work fine otherwise, but attaching volumes to running instances fails. The key error seems to be:

Failed to attach volume at mountpoint: /dev/vdb: libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'

But I've pasted the full trace through nova-compute.log on the unit hosting the instance in question:

https://pastebin.ubuntu.com/p/dQHFh9RBNs/

The charm is deployed with the following relations:

https://pastebin.ubuntu.com/p/mzx3KXMTJv/

[Barry Price (barryprice)]

This is using the stable/18.02 versions of the charms, btw.

[Barry Price (barryprice)]

Tracked this down to a cephx username mismatch. I've use bogus UUIDs here to illustrate.

Defined on the compute node:

$ sudo virsh secret-list
 UUID Usage
--------------------------------------------------------------------------------
 01234567-0123-4567-89ab-0123456789ab ceph client.nova-compute secret
 98765432-ba98-7654-3210-ba9876543210 ceph client.cinder-ceph secret

$

But we see logged in the XML from the attempted attach via nova-compute.log:

  <auth username="cinder">
    <secret type="ceph" uuid="98765432-ba98-7654-3210-ba9876543210"/>
  </auth>

Taking the (full) XML, from which the above is snipped, and amending username="cinder" to username="cinder-ceph" and applying this to "virsh attach-device" on the compute node results in the volume being successfully attached.

Alternatively, using the username "nova-compute" and its corresponding secret uuid also works.

[James Page (james-page)]

cinder should be providing the 'cinder-ceph' username as part of the block device setup - can you check that you don't have a direct relation between the cinder charm and ceph-mon as this might cause this problem

[Barry Price (barryprice)]

Ah, cinder is indeed related to ceph-mon, amongst others:

applications:
  cinder:
    charm: local:bionic/cinder-136
    series: bionic
    os: ubuntu
[snip]
    relations:
      amqp:
      - rabbitmq-server
      ceph:
      - ceph-mon
      cinder-volume-service:
      - nova-cloud-controller
      cluster:
      - cinder
      ha:
      - cinder-hacluster
      identity-service:
      - keystone
      image-service:
      - glance
      nrpe-external-master:
      - nrpe
      shared-db:
      - mysql
      storage-backend:
      - cinder-ceph

Should a simple 'juju remove-relation' do the right thing here, or is it going to be more complicated to recover from? We can redeploy the cloud if necessary.

[James Page (james-page)]

remove relation should do the right thing - but I'd remove any volumes that
have already been created first

we should drop that relation but the main issue is there is no nice
migration path for any older deployments which used the native ceph support
in the cinder charm.

On Mon, 19 Nov 2018 at 10:20 Barry Price <email address hidden> wrote:

> Ah, cinder is indeed related to ceph-mon, amongst others:
>
> applications:
> cinder:
> charm: local:bionic/cinder-136
> series: bionic
> os: ubuntu
> [snip]
> relations:
> amqp:
> - rabbitmq-server
> ceph:
> - ceph-mon
> cinder-volume-service:
> - nova-cloud-controller
> cluster:
> - cinder
> ha:
> - cinder-hacluster
> identity-service:
> - keystone
> image-service:
> - glance
> nrpe-external-master:
> - nrpe
> shared-db:
> - mysql
> storage-backend:
> - cinder-ceph
>
> Should a simple 'juju remove-relation' do the right thing here, or is it
> going to be more complicated to recover from? We can redeploy the cloud
> if necessary.
>
> --
> You received this bug notification because you are a member of OpenStack
> Charmers, which is subscribed to OpenStack nova-compute charm.
> Matching subscriptions: openstack-charms
> https://bugs.launchpad.net/bugs/1802910
>
> Title:
> Can't attach ceph volumes to instances on bionic/Queens deploy
>
> Status in OpenStack nova-compute charm:
> New
>
> Bug description:
> Creating instances and volumes seems to work fine otherwise, but
> attaching volumes to running instances fails. The key error seems to
> be:
>
> Failed to attach volume at mountpoint: /dev/vdb: libvirtError:
> internal error: unable to execute QEMU command 'device_add': Property
> 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
>
> But I've pasted the full trace through nova-compute.log on the unit
> hosting the instance in question:
>
> https://pastebin.ubuntu.com/p/dQHFh9RBNs/
>
> The charm is deployed with the following relations:
>
> https://pastebin.ubuntu.com/p/mzx3KXMTJv/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-nova-compute/+bug/1802910/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=charm-nova-compute; status=New;
> importance=Undecided; assignee=None;
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: barryprice james-page
> Launchpad-Bug-Reporter: Barry Price (barryprice)
> Launchpad-Bug-Modifier: Barry Price (barryprice)
> Launchpad-Message-Rationale: Subscriber (OpenStack nova-compute charm)
> @openstack-charmers
> Launchpad-Message-For: openstack-charmers
> Launchpad-Subscription: openstack-charms
>

[Barry Price (barryprice)]

Thanks, confirmed that removing the relation and starting again with a fresh volume/instance results in successful attach/detach.

Which I guess makes this a deployment error rather than a charm bug. We've fixed our spec that we deploy this cloud from to avoid running into this again, but I wonder whether this could/should be retargeted to nova-compute itself for at least better logging?

The errors in the initial pastebin above don't suggest any kind of authentication issue at all to me:

> Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'

[James Page (james-page)]

Repurposing this bug to deal with dropping the ceph support in the cinder charm.