Deployment fails if enable_outward_rabbitmq is set to 'yes'

[arlind]

Hi,

I have set enable_outward_rabbitmq: "yes" in globals.yml and deployment fails.

outward_rabbitmq docker log contains this error:

init terminating in do_boot ({error,{rabbitmq_prometheus,{bad_return,{{_},{_}}}}})

I have a 2 node baremetal cluster:

- os-controller01 (controller, network, monitoring)
- os-cs01 (compute, storage)

I have attached a zip file with logs and configuration:

- doutward_rabbitmq container logs - https://pastebin.com/pb7TXGNH
- ansible output - https://pastebin.com/gKRdW4Dg
- globals.yml - https://pastebin.com/DQmR3neL
- multinode inventory file - https://pastebin.com/X8mcfB8z
- /var/log/kolla/outward_rabbitmq directory files.

**Environment**:
* OS: CentOS Linux release 8.2.2004 (Core)
* Kernel: 4.18.0-193.el8.x86_64
* Docker version: Docker Engine - Community - 19.03.11
* Kolla-Ansible version: stable/ussuri branch
* Docker image Install type: binary
* Docker image distribution: centos

[arlind (arlind12)]

Apparently this bug comes from enabling rabbitmq_prometheus plugin, which listens on port 15692 and is not customizable. outward_rabbitmq fails to start because port 15692 is already used by the first rabbitmq container. This value is not configurable and is hardcoded in the config file '/var/lib/rabbitmq/mnesia/rabbit@<node_name>-plugins-expand/rabbitmq_prometheus-3.8.5/priv/schema/rabbitmq_prometheus.schema'

[arlind (arlind12)]

This is actually fixable by adding the following line in rabbitmq.conf:

prometheus.tcp.port = <port_number>

as suggetsed in the rabbitmq documentation: https://github.com/rabbitmq/rabbitmq-prometheus#configuration

I have prepared a pull request (gerrit change) to fix this issue:
https://review.opendev.org/#/c/738123/

[XiaojueGuan (xiaojuegaun)]

@arlind12 you may assign this bug to you and associate your commit https://review.opendev.org/#/c/738123/ with this bug by following this https://docs.openstack.org/contributors/code-and-documentation/using-gerrit.html#pushing-a-change

[Radosław Piliszek (yoctozepto)]

We are likely going to deprecate outward_rabbitmq with murano going away http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024120.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/821507
Committed: https://opendev.org/openstack/kolla-ansible/commit/1f3dcce5ac84414174f066dc20735e02ffebf041
Submitter: "Zuul (22348)"
Branch: master

commit 1f3dcce5ac84414174f066dc20735e02ffebf041
Author: LinPeiWen <email address hidden>
Date: Mon Dec 13 11:23:04 2021 +0800

    Support enable/disable rabbitmq prometheus plugins

    rabbitmq starting from 3.8.0, built-in Prometheus support,
    prometheus plugins are enabled by default, when the environment is
    "enable_prometheus is no", rabbitmq role will disable prometheus plugins

    Closes-Bug: #1885106

    Change-Id: I4d694d6224c813285d228d6bc7eece5731db1078

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/824018

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/824019

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/824020

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/824018
Committed: https://opendev.org/openstack/kolla-ansible/commit/cf8dbd6d08a8f47da62d1d4cd52ac5066e50380a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit cf8dbd6d08a8f47da62d1d4cd52ac5066e50380a
Author: LinPeiWen <email address hidden>
Date: Mon Dec 13 11:23:04 2021 +0800

    Support enable/disable rabbitmq prometheus plugins

    rabbitmq starting from 3.8.0, built-in Prometheus support,
    prometheus plugins are enabled by default, when the environment is
    "enable_prometheus is no", rabbitmq role will disable prometheus plugins

    Closes-Bug: #1885106

    Change-Id: I4d694d6224c813285d228d6bc7eece5731db1078
    (cherry picked from commit 1f3dcce5ac84414174f066dc20735e02ffebf041)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/824019
Committed: https://opendev.org/openstack/kolla-ansible/commit/b23bab245152840fca261d97680bb32e46d217d2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b23bab245152840fca261d97680bb32e46d217d2
Author: LinPeiWen <email address hidden>
Date: Mon Dec 13 11:23:04 2021 +0800

    Support enable/disable rabbitmq prometheus plugins

    rabbitmq starting from 3.8.0, built-in Prometheus support,
    prometheus plugins are enabled by default, when the environment is
    "enable_prometheus is no", rabbitmq role will disable prometheus plugins

    Closes-Bug: #1885106

    Change-Id: I4d694d6224c813285d228d6bc7eece5731db1078
    (cherry picked from commit 1f3dcce5ac84414174f066dc20735e02ffebf041)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/824020
Committed: https://opendev.org/openstack/kolla-ansible/commit/46544622ebf837b93f9f3b791440cf31b75a489a
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 46544622ebf837b93f9f3b791440cf31b75a489a
Author: LinPeiWen <email address hidden>
Date: Mon Dec 13 11:23:04 2021 +0800

    Support enable/disable rabbitmq prometheus plugins

    rabbitmq starting from 3.8.0, built-in Prometheus support,
    prometheus plugins are enabled by default, when the environment is
    "enable_prometheus is no", rabbitmq role will disable prometheus plugins

    Closes-Bug: #1885106

    Change-Id: I4d694d6224c813285d228d6bc7eece5731db1078
    (cherry picked from commit 1f3dcce5ac84414174f066dc20735e02ffebf041)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 11.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 12.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 13.1.0 release.