upgrades are being triggered and failing on fresh installs

[Ryan Beisner]

Ceph upgrades are being triggered and failing on fresh installs. This appears to have started with:

https://github.com/openstack/charm-ceph-osd/commit/a60775be188a6450f1ec3c8486aad92032acf1d0

The following bare metal tests are failing with this:
 - trusty-liberty
 - trusty-kilo

ceph-osd/0 error idle 1.25.9.1 0 table.maas hook failed: "config-changed"

[Ryan Beisner (1chb1n)]

2017-02-08 09:14:17 INFO install update-initramfs: Generating /boot/initrd.img-3.13.0-108-generic
2017-02-08 09:14:34 INFO juju-log Application Version: 0.94.8
2017-02-08 09:14:35 DEBUG juju-log Hardening function 'install'
2017-02-08 09:14:35 DEBUG juju-log Hardening function 'config_changed'
2017-02-08 09:14:35 DEBUG juju-log Hardening function 'upgrade_charm'
2017-02-08 09:14:35 DEBUG juju-log Hardening function 'update_status'
2017-02-08 09:14:35 DEBUG juju-log No hardening applied to 'config_changed'
2017-02-08 09:14:35 INFO juju-log old_version: firefly
2017-02-08 09:14:35 INFO juju-log new_version: hammer
2017-02-08 09:14:35 INFO juju-log firefly to hammer is a valid upgrade path. Proceeding.
2017-02-08 09:14:35 INFO juju-log roll_osd_cluster called with hammer
2017-02-08 09:14:35 INFO config-changed Error initializing cluster client: Error('error calling conf_read_file: errno EINVAL',)
2017-02-08 09:14:35 INFO juju-log ceph osd tree command failed with message:
2017-02-08 09:14:35 INFO config-changed Traceback (most recent call last):
2017-02-08 09:14:35 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 520, in <module>
2017-02-08 09:14:35 INFO config-changed hooks.execute(sys.argv)
2017-02-08 09:14:35 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-02-08 09:14:35 INFO config-changed self._hooks[hook_name]()
2017-02-08 09:14:35 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2017-02-08 09:14:35 INFO config-changed return f(*args, **kwargs)
2017-02-08 09:14:35 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 314, in config_changed
2017-02-08 09:14:35 INFO config-changed check_for_upgrade()
2017-02-08 09:14:35 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 108, in check_for_upgrade
2017-02-08 09:14:35 INFO config-changed upgrade_key='osd-upgrade')
2017-02-08 09:14:35 INFO config-changed File "lib/ceph/__init__.py", line 1527, in roll_osd_cluster
2017-02-08 09:14:35 INFO config-changed osd_tree = get_osd_tree(service=upgrade_key)
2017-02-08 09:14:35 INFO config-changed File "lib/ceph/__init__.py", line 467, in get_osd_tree
2017-02-08 09:14:35 INFO config-changed 'osd', 'tree', '--format=json'])
2017-02-08 09:14:35 INFO config-changed File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
2017-02-08 09:14:35 INFO config-changed raise CalledProcessError(retcode, cmd, output=output)
2017-02-08 09:14:35 INFO config-changed subprocess.CalledProcessError: Command '['ceph', '--id', 'osd-upgrade', 'osd', 'tree', '--format=json']' returned non-zero exit status 1
2017-02-08 09:14:35 ERROR juju.worker.uniter.operation runhook.go:107 hook "config-changed" failed: exit status 1

[Ryan Beisner (1chb1n)]

For context & bundle/config:

trusty-kilo and trusty-liberty targets:

http://bazaar.launchpad.net/~ost-maintainers/openstack-mojo-specs/mojo-openstack-specs/view/head:/helper/bundles/baremetal7-next.yaml#L40

[Ryan Beisner (1chb1n)]

FYI This is consistently reproducible on the metal mojo specs for the two targets listed (ie. not an occasional issue).

[Ryan Beisner (1chb1n)]

The initial proposed set of fixes [1] fail with:

2017-02-17 00:41:02 INFO config-changed Traceback (most recent call last):
2017-02-17 00:41:02 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 519, in <module>
2017-02-17 00:41:02 INFO config-changed hooks.execute(sys.argv)
2017-02-17 00:41:02 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-02-17 00:41:02 INFO config-changed self._hooks[hook_name]()
2017-02-17 00:41:02 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2017-02-17 00:41:02 INFO config-changed return f(*args, **kwargs)
2017-02-17 00:41:02 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 316, in config_changed
2017-02-17 00:41:02 INFO config-changed check_for_upgrade()
2017-02-17 00:41:02 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 92, in check_for_upgrade
2017-02-17 00:41:02 INFO config-changed if c.previous('source') == c('source') and c('source') is None:
2017-02-17 00:41:02 INFO config-changed TypeError: 'Config' object is not callable

[1] https://review.openstack.org/#/q/topic:bug/1662943

Also see attachment.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/435086
Committed: https://git.openstack.org/cgit/openstack/charm-ceph/commit/?id=d3cf8bb3aba40f588de63a13531a4d36c84dc09a
Submitter: Jenkins
Branch: master

commit d3cf8bb3aba40f588de63a13531a4d36c84dc09a
Author: Chris MacNaughton <email address hidden>
Date: Thu Feb 16 14:59:46 2017 -0500

    Only check for upgrades if bootstrapped

    Only check for upgrade requests if the local unit is installed
    and bootstrapped, avoiding attempts to upgrade on initial
    execution of config-changed for trusty UCA pockets.

    Note that the upgrade process relies on a running ceph cluster.

    Change-Id: Ia3efe2f8cfdac4317809681e7d169725c6bd9ef2
    Closes-Bug: 1662943

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/435088
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=9a5a710a1b1899ae6dacfe89a6eebe70bd8037e6
Submitter: Jenkins
Branch: master

commit 9a5a710a1b1899ae6dacfe89a6eebe70bd8037e6
Author: Chris MacNaughton <email address hidden>
Date: Thu Feb 16 15:00:20 2017 -0500

    Only check for upgrades if bootstrapped

    Only check for upgrade requests if the local unit is installed
    and bootstrapped, avoiding attempts to upgrade on initial
    execution of config-changed for trusty UCA pockets.

    Note that the upgrade process relies on a running ceph cluster.

    Change-Id: Ic7e427368a373ed853111d837a0223a75b46ce8e
    Closes-Bug: 1662943

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/435087
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=5e1ffebc94f82a4419a7c87b6ff657d5098b5410
Submitter: Jenkins
Branch: master

commit 5e1ffebc94f82a4419a7c87b6ff657d5098b5410
Author: Chris MacNaughton <email address hidden>
Date: Thu Feb 16 15:00:09 2017 -0500

    Only check for upgrades if bootstrapped

    Only check for upgrade requests if the local unit is installed
    and bootstrapped, avoiding attempts to upgrade on initial
    execution of config-changed for trusty UCA pockets.

    Note that the upgrade process relies on a running ceph cluster.

    Change-Id: Ia3cfcedc6bdc4aff3f50f1bba8f524ca850bbf12
    Closes-Bug: 1662943