Latest rabbitmq centos image crashes repeatedly

[Jim Rollenhagen]

[root@localhost kolla-ansible]# uname -a
Linux localhost 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost kolla-ansible]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@localhost kolla-ansible]# docker --version
Docker version 1.12.0, build 8eab29e
[root@localhost kolla-ansible]# docker images | grep rabbit
kolla/centos-source-rabbitmq master 86cc98d7ccd7 13 hours ago 516.2 MB

Rabbitmq crashes almost instantly every time the container starts, and it just spins and keeps trying to restart. Configs generated by kolla below.

[root@localhost kolla-ansible]# cat /etc/kolla/rabbitmq/rabbitmq.config
[
  {kernel, [
    {inet_dist_use_interface, {192,168,106,155}},
    {inet_dist_listen_min, 25672},
    {inet_dist_listen_max, 25672}
  ]},
  {rabbit, [
    {tcp_listeners, [
      {"192.168.106.155", 5672}
    ]},
    {cluster_partition_handling, autoheal}
  ]},
  {rabbitmq_management, [
    {listener, [
      {ip, "192.168.106.155"},
      {port, 15672}
    ]},
    {load_definitions, "/etc/rabbitmq/definitions.json"}
  ]},
  {rabbitmq_clusterer, [{config, "/etc/rabbitmq/rabbitmq-clusterer.config"}]}
].
% EOF
[root@localhost kolla-ansible]# cat /etc/kolla/rabbitmq/rabbitmq-env.conf
RABBITMQ_NODENAME=rabbit@localhost
RABBITMQ_BOOT_MODULE=rabbit_clusterer
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6/plugins/rabbitmq_clusterer-3.6.x.ez/rabbitmq_clusterer-3.6.x-667f92b0/ebin"

RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq

# TODO(sdake, vhosakot)
# erlang by default binds to wildcard (all interfaces) and can potentially
# interfere with the neutron external or tenant networks. We should in theory
# bind epmd to the host's IPv4 address to address the issue however this also
# has issues and can crash erlang when it is compiled with IPv6 support.
# See bugs:
# https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1374109
# https://bugs.launchpad.net/kolla/+bug/1562701
# https://bugzilla.redhat.com/show_bug.cgi?id=1324922
#export ERL_EPMD_ADDRESS=192.168.106.155
export ERL_EPMD_PORT=4369
[root@localhost kolla-ansible]# cat /etc/kolla/rabbitmq/rabbitmq-clusterer.config
[
  {version, 1},
  {nodes, [
      {'rabbit@localhost', disc} ]},
  {gospel,
    {node, 'rabbit@localhost'}}
].
[root@localhost kolla-ansible]# cat /etc/kolla/rabbitmq/config.json
{
    "command": "/usr/sbin/rabbitmq-server",
    "config_files": [
        {
            "source": "/var/lib/kolla/config_files/rabbitmq-env.conf",
            "dest": "/etc/rabbitmq/rabbitmq-env.conf",
            "owner": "rabbitmq",
            "perm": "0600"
        },
        {
            "source": "/var/lib/kolla/config_files/rabbitmq.config",
            "dest": "/etc/rabbitmq/rabbitmq.config",
            "owner": "rabbitmq",
            "perm": "0600"
        },
        {
            "source": "/var/lib/kolla/config_files/rabbitmq-clusterer.config",
            "dest": "/etc/rabbitmq/rabbitmq-clusterer.config",
            "owner": "rabbitmq",
            "perm": "0600"
        },
        {
            "source": "/var/lib/kolla/config_files/definitions.json",
            "dest": "/etc/rabbitmq/definitions.json",
            "owner": "rabbitmq",
            "perm": "0600"
        }
    ],
    "permissions": [
        {
            "path": "/var/lib/rabbitmq",
            "owner": "rabbitmq:rabbitmq",
            "recurse": true
        },
        {
            "path": "/var/log/kolla/rabbitmq",
            "owner": "rabbitmq:rabbitmq",
            "recurse": true
        }
    ]
}
[root@localhost kolla-ansible]# cat /etc/kolla/rabbitmq/definitions.json
{
  "vhosts": [
    {"name": "/"} ],
  "users": [
    {"name": "openstack", "password": "password", "tags": "administrator"} ],
  "permissions": [
    {"user": "openstack", "vhost": "/", "configure": ".*", "write": ".*", "read": ".*"} ],
  "policies":[
    {"vhost": "/", "name": "ha-all", "pattern": ".*", "apply-to": "all", "definition": {"ha-mode":"all"}, "priority":0} ]
}

[Jim Rollenhagen (jim-rollenhagen)]

Attached the crash dump

[Mark Goddard (mgoddard)]

Hi Jim, I suspect you're missing this kolla-ansible patch: https://review.openstack.org/#/c/584427/. Another thing to watch out for is that the hostname resolves to the IP on api_interface.

[Jim Rollenhagen (jim-rollenhagen)]

Aha, yes, that's it. Which stems from a mistake I made, where I used "pip install kolla-ansible" instead of "pip install ./kolla-ansible", when running from source.

So, it seems as if this is still a valid bug when running against the latest release, would you agree?

Thanks Mark :)

[Mark Goddard (mgoddard)]

The latest release is 6.1.0, in the Queens series. To use this you should
be using a set of compatible images kolla from the same release. Kolla
ansible should be able to detect which image tag to use, but perhaps your
config overrides this?

On Mon, 8 Oct 2018 at 14:15, Jim Rollenhagen <email address hidden> wrote:

> Aha, yes, that's it. Which stems from a mistake I made, where I used
> "pip install kolla-ansible" instead of "pip install ./kolla-ansible",
> when running from source.
>
> So, it seems as if this is still a valid bug when running against the
> latest release, would you agree?
>
> Thanks Mark :)
>
> --
> You received this bug notification because you are a member of kolla-
> drivers, which is subscribed to kolla.
> Matching subscriptions: kolla-bug
> https://bugs.launchpad.net/bugs/1796394
>
> Title:
> Latest rabbitmq centos image crashes repeatedly
>
> Status in kolla:
> New
>
> Bug description:
> [root@localhost kolla-ansible]# uname -a
> Linux localhost 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
> [root@localhost kolla-ansible]# cat /etc/redhat-release
> CentOS Linux release 7.5.1804 (Core)
> [root@localhost kolla-ansible]# docker --version
> Docker version 1.12.0, build 8eab29e
> [root@localhost kolla-ansible]# docker images | grep rabbit
> kolla/centos-source-rabbitmq master
> 86cc98d7ccd7 13 hours ago 516.2 MB
>
> Rabbitmq crashes almost instantly every time the container starts, and
> it just spins and keeps trying to restart. Configs generated by kolla
> below.
>
> [root@localhost kolla-ansible]# cat /etc/kolla/rabbitmq/rabbitmq.config
> [
> {kernel, [
> {inet_dist_use_interface, {192,168,106,155}},
> {inet_dist_listen_min, 25672},
> {inet_dist_listen_max, 25672}
> ]},
> {rabbit, [
> {tcp_listeners, [
> {"192.168.106.155", 5672}
> ]},
> {cluster_partition_handling, autoheal}
> ]},
> {rabbitmq_management, [
> {listener, [
> {ip, "192.168.106.155"},
> {port, 15672}
> ]},
> {load_definitions, "/etc/rabbitmq/definitions.json"}
> ]},
> {rabbitmq_clusterer, [{config,
> "/etc/rabbitmq/rabbitmq-clusterer.config"}]}
> ].
> % EOF
> [root@localhost kolla-ansible]# cat
> /etc/kolla/rabbitmq/rabbitmq-env.conf
> RABBITMQ_NODENAME=rabbit@localhost
> RABBITMQ_BOOT_MODULE=rabbit_clusterer
> RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa
> /usr/lib/rabbitmq/lib/rabbitmq_server-3.6/plugins/rabbitmq_clusterer-3.6.x.ez/rabbitmq_clusterer-3.6.x-667f92b0/ebin"
>
> RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq
>
> # TODO(sdake, vhosakot)
> # erlang by default binds to wildcard (all interfaces) and can
> potentially
> # interfere with the neutron external or tenant networks. We should in
> theory
> # bind epmd to the host's IPv4 address to address the issue however this
> also
> # has issues and can crash erlang when it is compiled with IPv6 support.
> # See bugs:
> # https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1374109
> # https://bugs.launchpad.net/kolla/+bug/1562701
> # https://bugzilla.redhat...

[Jim Rollenhagen (jim-rollenhagen)]

Ah, you're probably right. I have:

kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "master"

(As the intention was to run master).

Thanks! I'll close this out.