ceph-osd fails to pause: missing keyring file

[Ursula Junque]

Using cs:trusty/ceph-osd-235.

I have an HA cloud, mitaka, with 6 ceph-osd units. When trying to pause any of them I get:

message: 'Action pause failed: Command ''[''ceph'', ''osd'', ''out'', ''5'']'' returned
  non-zero exit status 1'
status: failed

juju debug-log shows it's unable to authenticate to cluster because a keyring file is missing:
https://pastebin.canonical.com/160892/
"auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring: (2) No such file or directory"

[Chris Holcombe (xfactor973)]

That's interesting that it's trying to look for the admin keyring. The osd charm and server don't have that. Hence the ceph osd out command failing.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/342437

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/342432
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=56828938caf007d558e7c333cc58185510738c40
Submitter: Jenkins
Branch: master

commit 56828938caf007d558e7c333cc58185510738c40
Author: Chris Holcombe <email address hidden>
Date: Thu Jul 14 14:01:30 2016 -0700

    Update cephx permissions for OSD actions

    This change adds more permissions to the ceph osd-upgrade
    key, specifically the ability to mark an OSD as in/out and
    actively remove an OSD from the cluster.

    Some of the ceph-osd charm maintenance actions are failing because
    of this lack of permission.

    Closes-Bug: 1602826
    Change-Id: I6af43b61149c6eeeeb5c77950701194beda2da71

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/343071
Committed: https://git.openstack.org/cgit/openstack/charm-ceph/commit/?id=becf5ef80f9ce0e6dcc5f5a02a91c1085b35a6c3
Submitter: Jenkins
Branch: master

commit becf5ef80f9ce0e6dcc5f5a02a91c1085b35a6c3
Author: Chris Holcombe <email address hidden>
Date: Fri Jul 15 13:47:42 2016 -0700

    Use osd-upgrade user for pause/resume

    The pause and resume actions shell out to the ceph command to run
    OSD operations (in/out).

    Because the default cephx key given out by the monitor cluster does
    not contain the correct permissions, these commands fail.

    Use the osd-upgrade user which has the correct permissions.

    Closes-Bug: 1602826

    Depends-On: I6af43b61149c6eeeeb5c77950701194beda2da71
    Change-Id: I95bedcdea622fbf2fd799e63932cedd0d577568a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/342436
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=25a988b9c4146485cc77e1d72f89086a4d4aafeb
Submitter: Jenkins
Branch: master

commit 25a988b9c4146485cc77e1d72f89086a4d4aafeb
Author: Chris Holcombe <email address hidden>
Date: Thu Jul 14 14:07:01 2016 -0700

    Fix OSD replacement

    Use the osd-upgrade key when replacing OSD's as this key
    has the correct cephx permissions to perform the operation.

    Closes-Bug: 1602826

    Depends-On: I6af43b61149c6eeeeb5c77950701194beda2da71
    Change-Id: I32d2f1a4036e09d5d1fd13009c95ab1514e7304c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/342437
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=3ab6133c2ad1a52c5188cb4236dd2f44448ea9d5
Submitter: Jenkins
Branch: master

commit 3ab6133c2ad1a52c5188cb4236dd2f44448ea9d5
Author: Chris Holcombe <email address hidden>
Date: Thu Jul 14 13:49:18 2016 -0700

    Use osd-upgrade user for pause/resume

    The pause and resume actions shell out to the ceph command to run
    OSD operations (in/out).

    Because the default cephx key given out by the monitor cluster does
    not contain the correct permissions, these commands fail.

    Use the osd-upgrade user which has the correct permissions.

    Closes-Bug: 1602826

    Depends-On: I6af43b61149c6eeeeb5c77950701194beda2da71
    Change-Id: Ie31bc9048972dbb0986ac8deb5b821a4db5d585f