ceph-fs amulet smoke failing on config file validation

[Ryan Beisner]

ceph-fs amulet smoke is consistently failing on:

DEBUG:runner:ceph config error: section [global] is missing option keyring

https://review.openstack.org/#/c/507579/

DEBUG:runner:call ['/tmp/bundletester-eU9ZrQ/ceph-fs/tests/gate-basic-xenial-mitaka'] (cwd: /tmp/bundletester-eU9ZrQ/ceph-fs)
DEBUG:runner:2017-09-26 21:24:02,835 __init__ INFO: OpenStackAmuletDeployment: init
DEBUG:runner:2017-09-26 21:24:02,835 _add_services INFO: OpenStackAmuletDeployment: adding services
DEBUG:runner:2017-09-26 21:24:02,835 _determine_branch_locations INFO: OpenStackAmuletDeployment: determine branch locations
DEBUG:runner:2017-09-26 21:24:12,361 _configure_services INFO: OpenStackAmuletDeployment: configure services
DEBUG:runner:2017-09-26 21:24:12 Starting deployment of auto-osci-sv10:admin/auto-osci-sv10
DEBUG:runner:2017-09-26 21:24:13 Deploying applications...
DEBUG:runner:2017-09-26 21:24:13 Deploying application ceph-fs using /tmp/charmLSA05h/xenial/ceph-fs
DEBUG:runner:2017-09-26 21:24:15 Deploying application ceph-mon using cs:~openstack-charmers-next/ceph-mon-292
DEBUG:runner:2017-09-26 21:24:18 Deploying application ceph-osd using cs:~openstack-charmers-next/ceph-osd-293
DEBUG:runner:2017-09-26 21:36:37 Adding relations...
DEBUG:runner:2017-09-26 21:36:37 Adding relation ceph-osd:mon <-> ceph-mon:osd
DEBUG:runner:2017-09-26 21:36:37 Adding relation ceph-fs:ceph-mds <-> ceph-mon:mds
DEBUG:runner:2017-09-26 21:39:47 Deployment complete in 935.03 seconds
DEBUG:runner:2017-09-26 21:40:15,285 __init__ INFO: Waiting on extended status checks...
DEBUG:runner:2017-09-26 21:40:15,286 _auto_wait_for_status INFO: Waiting for extended status on units...
DEBUG:runner:2017-09-26 21:40:15,286 _auto_wait_for_status DEBUG: Default extended status wait match: contains READY (case-insensitive)
DEBUG:runner:2017-09-26 21:40:15,287 _auto_wait_for_status DEBUG: Waiting up to 1800s for extended status on services: ['ceph-mon', 'ceph-fs', 'ceph-osd']
DEBUG:runner:2017-09-26 21:40:15,589 _auto_wait_for_status INFO: OK
DEBUG:runner:2017-09-26 21:40:48,594 validate_unit_process_ids DEBUG: Checking units for running processes...
DEBUG:runner:2017-09-26 21:40:48,597 validate_unit_process_ids DEBUG: Expected PIDs: {<amulet.sentry.UnitSentry object at 0x7fdf9a2c3610>: {'ceph-osd': 2}, <amulet.sentry.UnitSentry object at 0x7fdf9aa23b10>: {'ceph-mon': 1}, <amulet.sentry.UnitSentry object at 0x7fdf9a2c3090>: {'ceph-osd': 2}, <amulet.sentry.UnitSentry object at 0x7fdf9aa238d0>: {'ceph-mon': 1}, <amulet.sentry.UnitSentry object at 0x7fdf9a2c38d0>: {'ceph-mds': 1}, <amulet.sentry.UnitSentry object at 0x7fdf9aa23310>: {'ceph-mon': 1}, <amulet.sentry.UnitSentry object at 0x7fdf9a2c3190>: {'ceph-osd': 2}}
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: Actual PIDs: {<amulet.sentry.UnitSentry object at 0x7fdf9a2c3610>: {'ceph-osd': ['18259', '17611']}, <amulet.sentry.UnitSentry object at 0x7fdf9aa23b10>: {'ceph-mon': ['16746']}, <amulet.sentry.UnitSentry object at 0x7fdf9a2c3090>: {'ceph-osd': ['18240', '17547']}, <amulet.sentry.UnitSentry object at 0x7fdf9a2c38d0>: {'ceph-mds': ['16189']}, <amulet.sentry.UnitSentry object at 0x7fdf9aa238d0>: {'ceph-mon': ['16949']}, <amulet.sentry.UnitSentry object at 0x7fdf9aa23310>: {'ceph-mon': ['17721']}, <amulet.sentry.UnitSentry object at 0x7fdf9a2c3190>: {'ceph-osd': ['18421', '17631']}}
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: PID check OK: ceph-osd/0 ceph-osd 2: ['18259', '17611']
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: PID check OK: ceph-mon/2 ceph-mon 1: ['16746']
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: PID check OK: ceph-osd/2 ceph-osd 2: ['18240', '17547']
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: PID check OK: ceph-mon/1 ceph-mon 1: ['16949']
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: PID check OK: ceph-fs/0 ceph-mds 1: ['16189']
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: PID check OK: ceph-mon/0 ceph-mon 1: ['17721']
DEBUG:runner:2017-09-26 21:40:48,598 validate_unit_process_ids DEBUG: PID check OK: ceph-osd/1 ceph-osd 2: ['18421', '17631']
DEBUG:runner:2017-09-26 21:40:48,598 validate_services_by_name DEBUG: Checking status of system services...
DEBUG:runner:2017-09-26 21:40:48,598 test_300_ceph_config DEBUG: Checking ceph config file data...
DEBUG:runner:2017-09-26 21:40:48,598 validate_config_data DEBUG: Validating config file data (mds in /etc/ceph/ceph.conf on ceph-mon/0)...
DEBUG:runner:2017-09-26 21:40:53,834 validate_config_data DEBUG: Validating config file data (global in /etc/ceph/ceph.conf on ceph-mon/0)...
DEBUG:runner:ceph config error: section [global] is missing option keyring
DEBUG:runner:Exit Code: 1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/507839

[Chris MacNaughton (chris.macnaughton)]

The global keyring was removed from ceph-mon configuration to better support colocating ceph-mon and ceph-osd in the change at https://review.openstack.org/#/c/504114/.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507839
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-fs/commit/?id=502e84f7fd8a7d7e8c93feff5d069addfd9bc32f
Submitter: Jenkins
Branch: master

commit 502e84f7fd8a7d7e8c93feff5d069addfd9bc32f
Author: Chris MacNaughton <email address hidden>
Date: Wed Sep 27 08:40:57 2017 -0400

    Remove keyring from expected ceph.conf

    The global keyring was removed from ceph-mon configuration
    to better support colocating ceph-mon and ceph-osd.

    Change-Id: Ic1ce5f7fa4f0ba7f36b96b3204634b010d1346d3
    Closes-Bug: #1719869

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.08
Review: https://review.openstack.org/507859

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507859
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-fs/commit/?id=44093c736a30be74d048d72e8b5ba1956d93a51b
Submitter: Jenkins
Branch: stable/17.08

commit 44093c736a30be74d048d72e8b5ba1956d93a51b
Author: Chris MacNaughton <email address hidden>
Date: Wed Sep 27 08:40:57 2017 -0400

    Remove keyring from expected ceph.conf

    The global keyring was removed from ceph-mon configuration
    to better support colocating ceph-mon and ceph-osd.

    Change-Id: Ic1ce5f7fa4f0ba7f36b96b3204634b010d1346d3
    Closes-Bug: #1719869
    (cherry picked from commit 502e84f7fd8a7d7e8c93feff5d069addfd9bc32f)