rabbitmq setup from secondary IP

[Sebastien Fuchs]

If setup network is not the main interface (ie the same as ansible one), rabbitmq fails, even if the IP can be resolved:
"Hostname has to resolve to IP address of api_interface".

Ex:
network_interface: "eth1"

Analysis:
ansible/roles/rabbitmq/templates/rabbitmq_clusterer.config.j2 shows:
[
  {version, 1},
  {nodes, [
{% for host in groups['rabbitmq'] %} {'rabbit@{{ hostvars[host]['ansible_hostname'] }}', disc}{% if not loop.last %},{% endif %}

{% endfor %}
  ]},
  {gospel, {node, 'rabbit@{{ hostvars[groups['rabbitmq'][0]]['ansible_hostname'] }}'}}
].

Where {{ hostvars[host]['ansible_hostname'] }} resolves to the host running ansible (eth0).
It should resolve to the hostname attached to the IP of {{ network_interface }}.

[Jeffrey Zhang (jeffrey4l)]

what's your api_interface value in globals.yml file?

In kolla, rabbitmq connect each other through api_interface, so in hosts file or dns server, the hostname name should be resolved to the {{ api_interface }} address.

[bjolo (bjorn-lofdahl)]

hi all,

I also consider this to be a bug. From an operator perspective, hostname will/should be resolved to the IP of the network_interface and not the API interface. In openstack, the API interface (or the internal interface as it is normally called) is an isolated network w/o a gateway. Thus it does not seem correct that the DNS or /etc/hosts resolution should point to the API interface.

As it is today, rabbitmq requires hostnames to work (it does not work with IPs for some reason). Kolla solves this by populating the hosts /etc/hosts file or relies on external DNS. Arguably this is not the correct solution. DNS will most likely point to the network_interface, and populating /etc/hosts might not be working for several reasons. Order in nsswitch.conf matters for once, and depending on operator organization /etc/hosts can be under the control of some other CM system and any changes will be reverted. Another aspect is that on minimalistic OS's, modifying local root fs is not possible. At least not dynamically. Remember that in kolla it should be simple to move/remove/add rabbitmq containers. Thus the solution needed for this is better if it is contained within kolla and not require any external dependencies.

One solution for this could be that the rabbitmq containers generates their own /etc/hosts file in the containers with the API interface IP and some constructed hostname. Remember this is only needed for rabbitmq clustering so it does not really matter what the hostnames are, as long as they are resolvable and points to the right IP/interface.

ex
hostnamames are node-a, node-b, node-c and DNS resolution points to network_interface

in the rabbitmq containers, the /etc/hosts file looks something like below where 10.0.0.0/24 is the api network.
10.0.0.10 node-a-api
10.0.0.11 node-b-api
10.0.0.12 node-c-api

[Jeffrey Zhang (jeffrey4l)]

sound a good idea. Could u push a PS for this?

[Jeffrey Zhang (jeffrey4l)]

Kolla is using --net host solution, so the hostname is not changeable.

another solution is use --add-host <hostname>:<ip> parameter, but it will overwrite all the host os's /etc/hosts file. not optimal too.

btw, here the official explanation about why rabbitmq requires hostname resolve.

[0] http://www.rabbitmq.com/networking.html#dns

[Vladislav Belogrudov (vlad-belogrudov)]

I would suggest using simple dns containers as described in https://blueprints.launchpad.net/kolla/+spec/kolla-dns

[Vladislav Belogrudov (vlad-belogrudov)]

if dns containers are implemented one could write ansible for rabbits like this:

 'rabbit@rabbit1','rabbit@rabbit2','rabbit@rabbit3' up to max_rabbits defined

[Satya Sanjibani Routray (satroutr)]

I have the fix ready with me

Will be pushing the changes

[Christian Berendt (berendt)]

satroutr, the bug will be assigned to you after you pushed your review on gerrit.

[Satya Sanjibani Routray (satroutr)]

https://review.openstack.org/#/c/404944