charm does not remove old ceph config and packages (no -broken hook)

[Lei Wang]

If there was something wrong with ceph and other services (e.g. cinder, glance and nova-compute etc), and one destroys the relation between them, the old ceph config, ceph key and ceph packages are still in the system even the relation was destroyed.

When re-add relation between cinder/glance and ceph, the old ceph key is still there, and which causes wrong ceph key in cinder/glance node, and it will fail to contact with ceph

[James Page (james-page)]

Both these charms lack a -broken hook for the ceph relation; marking confirmed and medium for now.

[Liang Chen (cbjchen)]

Ray,
Ceph node doesn't delete keys upon a broken relation, and when the related service rejoins the relation with ceph, the old keys should still be valid. Can you please give me some detailed information on how this was produced? Did you manually removed the keys when fixed the problem with ceph?

[Liang Chen (cbjchen)]

s/a broken relation/a broken client_relation

[Lei Wang (raywang)]

Hi Liang,

When the related services rejoins the relations with ceph, the old keys might not be valid anymore, I guess ceph will recreate keys instead.

You might probably reproduce this by:

1. juju deploy ceph
2. juju deploy glance
3. juju add-relation ceph glance
4. juju destroy-relation ceph glance
5. juju add-relation ceph glance
6. juju ssh ceph/0
7. sudo ceph -s

You might not be able to check the ceph cluster since the keys are not correct.

[Liang Chen (cbjchen)]

Ray,
I still cannot reproduce this. Can you expand on "the old keys might not be valid"? AFAICT ceph charm doesn't remove keys upon broken client relation. In fact, ceph charm doesn't have a client-relation-departed hook. And next time, the client rejoins the relation with ceph, the charm client-relation-joined hook will try "ceph ... auth get-or-create <service>" key, which means if a key was created before for a service it will be simply returned again.

Below is the bundle I created for testing.

openstack-services:
  series: trusty
  services:
    mysql:
      branch: lp:charms/mysql
      constraints: mem=1G
      options:
        dataset-size: 50%
    rabbitmq-server:
      branch: lp:charms/rabbitmq-server
      constraints: mem=1G
    ceph:
      branch: lp:charms/ceph
      num_units: 2
      constraints: mem=1G
      options:
        monitor-count: 2
        fsid: 6547bd3e-1397-11e2-82e5-53567c8d32dc
        monitor-secret: AQCXrnZQwI7KGBAAiPofmKEXKxu5bUzoYLVkbQ==
        osd-devices: /dev/vdb
        osd-reformat: "yes"
        ephemeral-unmount: /mnt
    keystone:
      branch: lp:charms/keystone
      constraints: mem=1G
      options:
        admin-password: openstack
        admin-token: ubuntutesting
    glance:
      branch: lp:charms/glance
      constraints: mem=1G
  relations:
    - [ keystone, mysql ]
    - [ glance, mysql ]
    - [ glance, keystone ]
    - [ glance, ceph ]

After juju destroy-relation & add-relation, an image was created successfully.

ubuntu@cbjchen-bastion:~/openstack-charm-testing$ juju destroy-relation ceph glance
ubuntu@cbjchen-bastion:~/openstack-charm-testing$ glance image-create --name cirros26 --disk-format qcow2 --container-format bare --is-public True --file ../images/cirros
Request returned failure status.
500 Internal Server Error
Failed to upload image 35b3f5f4-8250-49f8-af39-f97dd2722091
    (HTTP 500)
ubuntu@cbjchen-bastion:~/openstack-charm-testing$ juju add-relation ceph glance
ubuntu@cbjchen-bastion:~/openstack-charm-testing$ glance image-create --name cirros26 --disk-format qcow2 --container-format bare --is-public True --file ../images/cirros
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 133eae9fb1c98f45894a4e60d8736619 |
| container_format | bare |
| created_at | 2014-11-18T21:50:27 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | d008db19-8281-4394-9a58-47e179d7f858 |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros26 |
| owner | 092e164fbef64890ae933a988c57d023 |
| protected | False |
| size | 13200896 |
| status | active ...

[Lei Wang (raywang)]

HI Liang,

Try to make some errors while you're destroy-relation with ceph, or destroy a services which has relation to ceph.

In this case, the old ceph keys and configs are left in the ceph nodes, and you deploy a service (e.g. cinder), and add-relation to ceph, it will use invalid keys and configs

[Liang Chen (cbjchen)]

Ray,

Still succeeded. It has been three months since the bug was filed, I wonder if it's been fixed somewhere.
Can you please detail the steps on how you encountered this issue (together with the bundle yaml if possible)?

juju destroy-service glance
juju add-relation glance keystone
juju add-relation glance mysql
juju add-relation glance ceph
glance image-create --name cirros31 --disk-format qcow2 --container-format bare --is-public True --file ../images/cirros
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 133eae9fb1c98f45894a4e60d8736619 |
| container_format | bare |
| created_at | 2014-11-19T15:30:15 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 8948d00a-82fb-44c3-b65b-73fd8d4ec9b0 |
| is_public | True |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros31 |
| owner | 092e164fbef64890ae933a988c57d023 |
| protected | False |
| size | 13200896 |
| status | active |
| updated_at | 2014-11-19T15:30:16 |
| virtual_size | None |
+------------------+--------------------------------------+

[Liang Chen (cbjchen)]

OK. This is reproduced by destroying and adding back the ceph service.
glance charms will not store the new key if one exists.

[Liang Chen (cbjchen)]

Waiting on the merge request here - https://code.launchpad.net/~cbjchen/charm-helpers/delete-ceph-keyring as some of the change needs to be done in charm-helper.
Fix will be proposed after the merge request is approved and code is synced to nova-compute charm.

[Lei Wang (raywang)]

Hi Liang

just a reminder that this bug is not only affect on nova-compute charm, but also cinder, glance, and other charms which has the relation with ceph.

[Liang Chen (cbjchen)]

Ray,
Sure. After the charm-helper patch is merged, I will submit fix for each of those affected packages.