RabbitMQ should use predefined ports below ephemeral ports range

[Michele Baldessari]

Currently RabbitMQ cluster uses a predefined port 35672 for clustering. This port belongs to so-called ephemeral ports range.

Ephemeral ports are the ports kernel assigns to application if it doesn't specify which port to open. So there is a small chance that this application being started before RabbitMQ itself could grab this port. Unfortunately we've just saw this in the wild.

The error in the rabbit logs will be something like the following:
Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",eaddrinuse]} in /var/log/rabbitmq/startup_err

Which really means some other application opened port 35672.

If we need static predefined port, then we'd better use port 25672. It doesn't belong to ephemeral ports range, so chances are low that anyone opens this port by mistake.

[Michele Baldessari (michele)]

I am raising this bug to High because now that we landed HA-NG the chances that services start before rabbitmq will take the 35672 port in the ephemeral range is actually a lot higher because services can and will start before rabbitmq.

[Alex Schultz (alex-schultz)]

You can also just reserve the port using sysctl and setting net.ipv4.ip_local_reserved_ports to include the rabbitmq port. Given that the keystone port also falls in this situtation, we should probably be reserving these ports. In my experiance there's a few other openstack services that also do similar silly things so you could use 35357,41055-41056,49000-49001,55572,58882 as a starting point to reserve

[Michele Baldessari (michele)]

Working on this via https://review.openstack.org/#/c/345851/

Alex, do you know which other services do this ephemeral port silliness? Is it just the ones you listed above? I would not mind trying to fix them all if possible (if not, your suggestion via the sysctl is also fine by me)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/345851
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a1dcc16f3addd43c229553706b2ba080cde7ea31
Submitter: Jenkins
Branch: master

commit a1dcc16f3addd43c229553706b2ba080cde7ea31
Author: Michele Baldessari <email address hidden>
Date: Fri Jul 22 08:40:25 2016 +0200

    Move rabbit's clustering port away from the ephemeral port range

    Currently RabbitMQ cluster uses a predefined port 35672 for clustering.
    This port belongs to so-called ephemeral ports range.

    Ephemeral ports are the ports kernel assings to application if it
    doesn't specify which port to open. So there is a small chance that this
    application being started before RabbitMQ itself could grab this port.
    While rather unlikely we did see this happen.

    Selinux change should already be in place. On my Centos 7 we have:
    rabbitmq_port_t tcp 25672
    corenet_tcp_bind_rabbitmq_port(rabbitmq_t)
    corenet_tcp_connect_rabbitmq_port(rabbitmq_t)

    First noted via:
    https://bugzilla.redhat.com/show_bug.cgi?id=1357522

    Closes-Bug: #1623818

    Depends-On: I0bcd0d063a7a766483426fdd5ea81cbe1dfaa348
    Change-Id: I995bd96c2a17614e954ea5bbae4d58998ef420dc