leader-settings-changed hook sometimes runs out of order?

[David Britton]

When min_cluster_count is unset, sometimes, you get this failure. For me it tends to happen on the third unit to join.

unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:45 INFO unit.rabbitmq-server/2.install logger.go:40 Setting up python-wheel (0.24.0-1~ubuntu1) ...
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:45 INFO unit.rabbitmq-server/2.install logger.go:40 Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 Traceback (most recent call last):
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 811, in <module>
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 hooks.execute(sys.argv)
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/charmhelpers/core/hookenv.py", line 672, in execute
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 self._hooks[hook_name]()
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 788, in leader_settings_changed
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 update_cookie(leaders_cookie=cookie)
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/leader-settings-changed", line 383, in update_cookie
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 with open(rabbit.COOKIE_PATH, 'r') as f:
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 INFO unit.rabbitmq-server/2.leader-settings-changed logger.go:40 IOError: [Errno 2] No such file or directory: '/var/lib/rabbitmq/.erlang.cookie'
unit-rabbitmq-server-2[2099]: 2015-09-29 19:41:46 ERROR juju.worker.uniter.operation runhook.go:103 hook "leader-settings-changed" failed: exit status 1

[David Ames (thedac)]

This happens when leader-settings-changed executes before config-changed has run and rabbitmq-server has not yet been installed. The directory /var/lib/rabbitmq does not exist yet.