CentOS RabbitMQ Unsupported Erlang Version

[Joseph M]

What happened: The RabbitMQ version and its corresponding Erlang version are not compatible according to upstream documentation: https://www.rabbitmq.com/which-erlang.html

For example in Steins case it runs RabbitMQ 3.7.10 which should have at least Erlang 20.3.x. The version actually in the container image, provided by the RDO repo, is 19.3.6. Although it is hard to prove it could be contributing to strange RabbitMQ behavior seen by the community. In my case it can result in queues timing out during failover events:

Error: operation node_health_check on node rabbit@node1 timed out. Timeout value used: 70000

2019-11-19 09:59:48.568 6 ERROR oslo_service.service MessageDeliveryFailure: Unable to connect to AMQP server on 10.101.0.3:5672 after inf tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute.node5' in vhost '/' due to timeout

What you expected to happen: The Kolla image to provide a supported RabbitMQ/Erlang version combination.

How to reproduce it (minimal and precise): Build a kolla stein rabbitmq image.

**Environment**:
* OS (e.g. from /etc/os-release): CentOS Linux release 7.7.1908
* Kernel (e.g. `uname -a`): 3.10.0-957.21.3.el7.x86_64
* Docker version if applicable (e.g. `docker version`): 19.03.2
* Kolla version (e.g. `git head or tag or stable branch` or pip package version if using release): stable/stein 7e31dac6a3

[Joseph M (noxoid)]

It may be useful to add the following repo which is provided by the RabbitMQ team for CentOS: https://bintray.com/rabbitmq-erlang/rpm/erlang

[Radosław Piliszek (yoctozepto)]

Not sure about importance (medium chosen) but looks like we are running an unsupported build. Might explain why ipv6 is b0rken as well.
On Ubuntu it's 20.2 so it's at least matching the major, but still lacking the minor (20.3).
This might explain occasional CI hiccups with rmq but these are rarest (unless some other ones are hidden rmq issues, cannot be ruled out).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/695356

[Mark Goddard (mgoddard)]

I think we are reading this wrong. RabbitMQ 3.7.10 supports 19.3.x to 21.x according to https://www.rabbitmq.com/which-erlang.html. So the versions we use are compatible.

[Radosław Piliszek (yoctozepto)]

The page has changed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dincer Celik (<email address hidden>) on branch: master
Review: https://review.opendev.org/695356

[Radosław Piliszek (yoctozepto)]

The main assumption turned out to be false.