Comment 6 for bug 1835615

So the failure is on controller-2 which is the bootstrap node (since when is this the case I am not sure). My current theory is that the rabbitmq cluster gets bootstrapped on controller-0 and then spawns up on ctrl-1 and ctrl-2 and when we call the following snippet:
rabbitmq_policy { 'ha-all@/':
        applyto => 'queues',
        pattern => '^(?!amq\.).*',
        definition => hash($ha_policy.map |$k, $v| {[$k, "${v}"]}),
}

The rabbitmq provider barfs on the output because likely at some point the command has some unexpected output.

We do run that command *only* after this 'rabbitmqctl eval "rabbit_mnesia:is_clustered()." | grep -q true', so it might be that this condition is not enough. I for one was able to observe the output of list_policies changing during a restart:
 while true; do rabbitmqctl -q list_policies -p /; done
vhost name pattern apply-to definition priority
/ ha-all ^(?!amq\.).* all {"ha-mode":"exactly","ha-params":2,"ha-promote-on-shutdown":"always"} 0
vhost name pattern apply-to definition priority
/ ha-all ^(?!amq\.).* all {"ha-mode":"exactly","ha-params":2,"ha-promote-on-shutdown":"always"} 0
vhost name pattern apply-to definition priority
/ ha-all ^(?!amq\.).* all {"ha-mode":"exactly","ha-params":2,"ha-promote-on-shutdown":"always"} 0
vhost name pattern apply-to definition priority
/ ha-all ^(?!amq\.).* all {"ha-mode":"exactly","ha-params":2,"ha-promote-on-shutdown":"always"} 0
vhost name pattern apply-to definition priority
/ ha-all ^(?!amq\.).* queues {"ha-mode":"exactly","ha-params":2,"ha-promote-on-shutdown":"always"} 0
vhost name pattern apply-to definition priority
/ ha-all ^(?!amq\.).* queues {"ha-mode":"exactly","ha-params":2,"ha-promote-on-shutdown":"always"} 0
vhost name pattern apply-to definition priority

It is likely that when this happens puppet-rabbitmqctl barfs due to either some other unexpected output and/or this output changing under its feet ("all" vs "queues"). To be exactly sure we'd really need the --debug option to these containers (this used to work but somehow got broken in the past)