unable to connect to epmd (port 4369) on CentOS7

[Marcin Dulak]

openstack-ansible bbf804f43363d9a867b2fd7af995bd892f3fa746 on CentOS7.

`rabbitmqctl -q cluster_status` fails in rabbit-mq-container containers.

TASK [rabbitmq_server : Check cluster status] ******************************************************************************************************************************************
Thursday 25 January 2018 01:26:25 +0000 (0:00:00.298) 2:24:16.082 ******
fatal: [dev-os1-hci3_rabbit_mq_container-e3e45be1]: FAILED! => {"changed": false, "cmd": "rabbitmqctl -q cluster_status | grep '{cluster_name,<<\"openstack\">>}'", "delta": "0:00:00.704820", "end": "2018-01-25 01:26:26.349838", "msg": "non-zero return code", "rc": 1, "start": "2018-01-25 01:26:25.645018", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Entering the containers and checking the status manually shows:

[root@dev-os1-hci3-rabbit-mq-container-e3e45be1 ~]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@dev-os1-hci3'
Error: unable to connect to node 'rabbit@dev-os1-hci3': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@dev-os1-hci3']

rabbit@dev-os1-hci3:
  * unable to connect to epmd (port 4369) on dev-os1-hci3: address (cannot connect to host/port)

current node details:
- node name: 'rabbitmq-cli-71@dev-os1-hci3-rabbit-mq-container-e3e45be1'
- home dir: /var/lib/rabbitmq
- cookie hash: G7h7wHTwLotdZPIm7rMsVw==

As mentioned in several other bugs this has something to do with the hostname, since:

[root@dev-os1-hci3-rabbit-mq-container-e3e45be1 ~]# hostname
dev-os1-hci3-rabbit-mq-container-e3e45be1

[root@dev-os1-hci3-rabbit-mq-container-e3e45be1 ~]# cat /etc/hostname
dev-os1-hci3-rabbit-mq-container-e3e45be1

[root@dev-os1-hci3-rabbit-mq-container-e3e45be1 ~]# HOSTNAME=$(hostname) rabbitmqctl cluster_status
Cluster status of node 'rabbit@dev-os1-hci3-rabbit-mq-container-e3e45be1'
[{nodes,[{disc,['rabbit@dev-os1-hci1-rabbit-mq-container-90ca1526',
                'rabbit@dev-os1-hci2-rabbit-mq-container-672b294e',
                'rabbit@dev-os1-hci3-rabbit-mq-container-e3e45be1']}]},
 {running_nodes,['rabbit@dev-os1-hci1-rabbit-mq-container-90ca1526',
                 'rabbit@dev-os1-hci2-rabbit-mq-container-672b294e',
                 'rabbit@dev-os1-hci3-rabbit-mq-container-e3e45be1']},
 {cluster_name,<<"openstack">>},
 {partitions,[]},
 {alarms,[{'rabbit@dev-os1-hci1-rabbit-mq-container-90ca1526',[]},
          {'rabbit@dev-os1-hci2-rabbit-mq-container-672b294e',[]},
          {'rabbit@dev-os1-hci3-rabbit-mq-container-e3e45be1',[]}]}]

The ~/.bashrc file is the default:

[root@dev-os1-hci3-rabbit-mq-container-e3e45be1 ~]# cat ~/.bashrc
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
 . /etc/bashrc
fi

[Jean-Philippe Evrard (jean-philippe-evrard)]

What's in your /etc/bashrc?

Could you drop your openstack_user_config? It looks like you're trying to connect to rabbit@dev-os1-hci3 instead of rabbit@dev-os1-hci3-rabbit-mq-container-e3e45be1 , which makes me think of an issue with the inventory...

Could you also give us the openstack_inventory.json, please?

[Marcin Dulak (marcin-dulak)]

I see some containers in openstack_inventory.json use the IPs from used_ips.
I start doubting the consistency of those two files, but you may be able to spot some configuration problems anyway.

[Marcin Dulak (marcin-dulak)]

I don't encounter this problem anymore on a fresh installation of master

https://github.com/openstack/openstack-ansible/commit/e883d33630b06c921ec71b09169940dc7a2c6914
https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/22f663a9c3671bf6e7a7a8fd863bac8f2943e5e3

[Launchpad Janitor (janitor)]

[Expired for openstack-ansible because there has been no activity for 60 days.]