hacluster helper causes deployment failures if relations are made before config options are set

[Ryan Beisner]

The hacluster helper (and subsequently charms such as keystone), are not tolerant of missing cluster data. This causes them to fail a hook if the ha-relation-joined hook is hit before the charm config option is applied.

Instead of failing the deployment, the helper and affected charms should probably go into a blocked state, awaiting ample configuration information.

http://paste.ubuntu.com/14075570/

# Unit log & trace
# NOTE: In this output, I patched get_hacluster_config() in charmhelpers/contrib/hahelpers/cluster.py to spit out the missing data list along with the error.

unit-keystone-2[9632]: 2015-12-17 17:01:45 ERROR unit.keystone/2.juju-log server.go:268 ha:3: Insufficient config data to configure hacluster: ['vip']
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 Traceback (most recent call last):
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 File "/var/lib/juju/agents/unit-keystone-2/charm/hooks/ha-relation-joined", line 647, in <module>
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 main()
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 File "/var/lib/juju/agents/unit-keystone-2/charm/hooks/ha-relation-joined", line 639, in main
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 hooks.execute(sys.argv)
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 File "/var/lib/juju/agents/unit-keystone-2/charm/hooks/charmhelpers/core/hookenv.py", line 704, in execute
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 self._hooks[hook_name]()
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 File "/var/lib/juju/agents/unit-keystone-2/charm/hooks/ha-relation-joined", line 498, in ha_joined
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 cluster_config = get_hacluster_config()
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 File "/var/lib/juju/agents/unit-keystone-2/charm/hooks/charmhelpers/contrib/hahelpers/cluster.py", line 295, in get_hacluster_config
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 raise HAIncompleteConfig
unit-keystone-2[9632]: 2015-12-17 17:01:45 INFO unit.keystone/2.ha-relation-joined logger.go:40 charmhelpers.contrib.hahelpers.cluster.HAIncompleteConfig
unit-keystone-2[9632]: 2015-12-17 17:01:45 ERROR juju.worker.uniter.operation runhook.go:107 hook "ha-relation-joined" failed: exit status 1

# Juju stat (before charm config options were applied)

[Services]
NAME STATUS EXPOSED CHARM
hacluster false local:trusty/hacluster-0
keystone error false local:trusty/keystone-0
mysql unknown false local:trusty/mysql-326

[Units]
ID WORKLOAD-STATE AGENT-STATE VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
keystone/0 error idle 1.25.1 1 10.5.24.161 hook failed: "ha-relation-joined" for hacluster:ha
  hacluster/0 active idle 1.25.1 10.5.24.161 Unit is ready and clustered
keystone/1 error idle 1.25.1 2 10.5.24.162 hook failed: "ha-relation-joined" for hacluster:ha
  hacluster/1 active idle 1.25.1 10.5.24.162 Unit is ready and clustered
keystone/2 error idle 1.25.1 3 10.5.24.163 hook failed: "ha-relation-joined" for hacluster:ha
  hacluster/2 active idle 1.25.1 10.5.24.163 Unit is ready and clustered
mysql/0 unknown idle 1.25.1 4 10.5.24.164

[Machines]
ID STATE VERSION DNS INS-ID SERIES HARDWARE
0 started 1.25.1 10.5.24.160 3bbb58db-0cf5-4bc2-95c2-7df3e490f587 trusty arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
1 started 1.25.1 10.5.24.161 ac00dcdc-0abf-4b25-945c-12b0f42e7e7c trusty arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
2 started 1.25.1 10.5.24.162 87f381a2-efeb-4b71-916a-c53cbf31853f trusty arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
3 started 1.25.1 10.5.24.163 136322e1-296b-419a-9419-b2f625e7ba90 trusty arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova
4 started 1.25.1 10.5.24.164 d3cf34b4-5d89-42d4-8a5a-b16de1b21941 trusty arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M availability-zone=nova