Could not contact node / connection attempt from disallowed node on new installation

[Niels van Adrichem]

Using charm cs:rabbitmq-server revision 75 on xenial/queens cloud with only 1 rabbitmq-server node in the cluster, deployment of rabbitmq-server with all-default options gives an error.

After the message 'Enabling queue mirroring', the charm hangs with a message 'hook failed: "config-changed"'.

The Juju unit-log (find full log attached) indicates applying the plugin configuration fails:
2018-07-16 08:19:30 DEBUG config-changed Applying plugin configuration to rabbit@juju-a89d1d-0-lxd-2... failed.
2018-07-16 08:19:30 DEBUG config-changed * Could not contact node rabbit@juju-a89d1d-0-lxd-2.
2018-07-16 08:19:30 DEBUG config-changed Changes will take effect at broker restart.
2018-07-16 08:19:30 DEBUG config-changed * Options: --online - fail if broker cannot be contacted.
2018-07-16 08:19:30 DEBUG config-changed --offline - do not try to contact broker.

The rabbit access logs indicated it finds the connection attempt disallowed:
=ERROR REPORT==== 16-Jul-2018::08:19:30 ===
** Connection attempt from disallowed node 'rabbitmq-cli-5304@juju-a89d1d-0-lxd-2' **

RabbitMQs startup_log (startup_err is empty):
              RabbitMQ 3.5.7. Copyright (C) 2007-2015 Pivotal Software, Inc.
  ## ## Licensed under the MPL. See http://www.rabbitmq.com/
  ## ##
  ########## Logs: /<email address hidden>
  ###### ## /<email address hidden>
  ##########
              Starting broker... completed with 0 plugins.

Forward and Reverse DNS (both FQDN and search) give correct results and the hosts-file also contains the right hostname to IP address entry.

I suspect this may be related to the "bit racey" behaviour described in the README, however, running `juju resolved rabbitmq-server/0` does not solve the problem. What worked on 2 occassions is to `juju debug rabbitmq-server/0`, run `juju resolved rabbitmq-server/0` from another session, wait for a couple of minutes, exit the debug-session and set to resolved again and suddenly it picks up again. The charm rerunning the config-changed hook periodically, however, does not solve the problem.

Afterwards, the startup_log indicates the broker starts with 6 plugins and openstack installation finishes succesful.

[Niels van Adrichem (nvanadrichem)]

Additionally, last time the debug-hooks trick didn't work until I restarted the rabbitmq-server using `service rabbitmq-server restart`.

[Niels van Adrichem (nvanadrichem)]

In fact, I found this bug only occurs when the option management_plugin is set to true, when set to default/false the charm installs flawlessly. Also unsetting the option through `juju config rabbitmq-server --reset management_plugin` and then running `juju resolved rabbitmq-server/0` reverts to a working environment when already in error state.

[David Ames (thedac)]

Niels,

I am trying to re-create this bug. If you happen to catch it again please provide the full juju log: /var/log/juju/unit-rabbitmq-server-$N.log.

In the meantime, I am running tests. We do a lot of testing turning the management plugin on and off but not as much at deploy time. So I'll be testing that.

[Launchpad Janitor (janitor)]

[Expired for OpenStack rabbitmq-server charm because there has been no activity for 60 days.]