Yoga Openstack kolla rabbitmq 3.9.29 is crashing

[VO LE HUY]

Im using ubun 20.04, ansible core 2.12.10 and kolla ansible stable/yoga branch. Deploy gave me exception, but before it ok !

crasher:
  initial call: rabbit_reader:init/3
  pid: <0.40128.0>
  registered_name: []
  exception exit: {unexpected_message,{'EXIT',#Port<0.3919>,einval}}
    in function rabbit_reader:handle_other/2 (rabbit_reader.erl, line 641)
    in call from rabbit_reader:mainloop/4 (rabbit_reader.erl, line 532)
    in call from rabbit_reader:run/1 (rabbit_reader.erl, line 454)
    in call from rabbit_reader:start_connection/4 (rabbit_reader.erl, line 353)
  ancestors: [<0.40126.0>,<0.924.0>,<0.923.0>,<0.922.0>,<0.920.0>,
                <0.919.0>,rabbit_sup,<0.235.0>]
  message_queue_len: 0
  messages: []
  links: [<0.40126.0>]
  dictionary: [{process_name,
                    {rabbit_reader,
                        <<"192.168.10.38:55076 -> 192.168.10.38:5672 - nova-conductor:527:ad721106-052e-418f-938f-d9a70d1478dd">>}},
                {connection_user_provided_name,
                    <<"nova-conductor:527:ad721106-052e-418f-938f-d9a70d1478dd">>},
                {client_properties,
                    [{<<"product">>,longstr,<<"py-amqp">>},
                     {<<"product_version">>,longstr,<<"5.0.9">>},
                     {<<"capabilities">>,table,
                      [{<<"authentication_failure_close">>,bool,true},
                       {<<"connection.blocked">>,bool,true},
                       {<<"consumer_cancel_notify">>,bool,true}]},
                     {<<"connection_name">>,longstr,
                      <<"nova-conductor:527:ad721106-052e-418f-938f-d9a70d1478dd">>}]},
                {{ch_pid,<0.40137.0>},
                 {1,#Ref<0.846123449.1562902529.248729>}},
                {{channel,1},
                 {<0.40137.0>,{method,rabbit_framing_amqp_0_9_1}}}]
  trap_exit: true
  status: running
  heap_size: 6772
  stack_size: 28
  reductions: 12146
neighbours:

  supervisor: {<0.40126.0>,rabbit_connection_sup}
  errorContext: child_terminated
  reason: {unexpected_message,{'EXIT',#Port<0.3919>,einval}}
  offender: [{pid,<0.40128.0>},
             {id,reader},
             {mfargs,{rabbit_reader,start_link,
                                    [<0.40127.0>,
                                     {acceptor,{192,168,10,38},5672}]}},
             {restart_type,intrinsic},
             {shutdown,300000},
             {child_type,worker}]
  supervisor: {<0.40126.0>,rabbit_connection_sup}
  errorContext: shutdown
  reason: reached_max_restart_intensity
  offender: [{pid,<0.40128.0>},
             {id,reader},
             {mfargs,{rabbit_reader,start_link,
                                    [<0.40127.0>,
                                     {acceptor,{192,168,10,38},5672}]}},
             {restart_type,intrinsic},
             {shutdown,300000},
             {child_type,worker}]

[Chris Stacey (cstacey4444)]

Seeing the same thing,
Kolla-ansible : stable/yoga
RabbitMQ : 3.9.29

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[VO LE HUY (huyvl3)]

I just provided OS

[Lena Voytek (lvoytek)]

Hello, Thank you for the bug report.

Would you be willing to provide the set of commands you used to reproduce this issue?

[VO LE HUY (huyvl3)]

@Lena Voytek (lvoytek),

It just deployed, when rabbitmq crashed so the Nova, Cinder, .... will "unhealthy" and you can not create VM.

[VO LE HUY (huyvl3)]

@Lena Voytek (lvoytek),

You need to use
- Ubuntu 20.04
- Ansible core 2.12.10
- Kolla Ansible @stable/yoga

The commands bootstrap-server and prechecks ok, but deploy was failure.

[VO LE HUY (huyvl3)]

@Lena Voytek (lvoytek),

That is what I did

$ cat ./multinode
--------------------------------------------------------
[control]
con[1:3] ansible_connection=ssh ansible_user=root
#control[01:03] ansible_user=kolla
[network]
con[1:3] ansible_connection=ssh ansible_user=root
[compute]
com[1:3] ansible_connection=ssh ansible_user=root
[monitoring]
con[1:3] ansible_connection=ssh ansible_user=root
[storage]
con[1:3] ansible_connection=ssh ansible_user=root
[deployment]
localhost ansible_connection=local
--------------------------------------------------------

$ /etc/kolla/globals.yml
--------------------------------------------------------
kolla_base_distro: "ubuntu"
kolla_install_type: "binary"
openstack_release: "yoga"
kolla_internal_vip_address: "192.168.10.22"
network_interface: "eth0"
neutron_external_interface: "eth3"
nova_compute_virt_type: "qemu"
neutron_plugin_agent: "ovn"
enable_cinder_backend_iscsi: "yes"
enable_cinder: "yes"
enable_cinder_backend_lvm: "yes"
enable_cinder_backup: "no"
enable_masakari: "yes"
keepalived_virtual_router_id: "51"
--------------------------------------------------------

$ root@kolla:~# lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description: Ubuntu 20.04.5 LTS
  Release: 20.04
  Codename: focal
$ root@kolla:~# ansible --version
    /usr/lib/python3/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
    "class": algorithms.Blowfish,
    ansible [core 2.12.10]
    config file = /etc/ansible/ansible.cfg
    configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    ansible python module location = /usr/lib/python3/dist-packages/ansible
    ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
    executable location = /usr/bin/ansible
    python version = 3.8.10 (default, May 26 2023, 14:05:08) [GCC 9.4.0]
    jinja version = 3.1.2
    libyaml = True
$ root@kolla:~#
$ root@kolla:~#
$ root@kolla:~# kolla-ansible --version
  14.8.1
$root@kolla:~#

[VO LE HUY (huyvl3)]

@Chris Stacey (cstacey4444),

I just checked, the same with me

$ (rabbitmq)[root@kolla-controller /]# rabbitmqctl --version
  3.9.29

[Athos Ribeiro (athos-ribeiro)]

Hi VO LE HUY,

thanks for reporting this issue.

As Lena mentioned, could you provide a minimal reproducer for the issue? Ideally, it could be a script that will run on a fresh install and will result in the issue you are experiencing so we can track the actual fault down.

[Boris Lukashev (rageltman)]

We just upgraded our pre-production cloud from xena->yoga and this is happening mid upgrade, preventing completion and creating an indeterminate state.
Original deploy was 20.04 w/ Xena which worked fine a few hours ago and now does not (with those symptoms all over the place) after upgrading to yoga stable.

@Athos Ribeiro: 3 "control nodes" with 15 nova/zun/ovn nodes
Image is now https://quay.io/repository/openstack.kolla/ubuntu-source-rabbitmq/manifest/sha256:40e788bb4122fd9546a7f9c8ebee48c8a4de20cdd2673cc9c7767f720261018e

Resetting queues in rabbitmq does not help, nor does enabling om_enable_rabbitmq_high_availability and going through the reconfiguration procedure. Looks like an actual bug in the container.

[VO LE HUY (huyvl3)]

@Athos Ribeiro (athos-ribeiro),

Sure, it just

$ kolla-ansible -i multinode bootstrap-servers
$ kolla-ansible -i multinode prechecks
$ kolla-ansible -i multinode deploy

[VO LE HUY (huyvl3)]

@Boris Lukashev (rageltman),

In my case, Rabbitmq log spamed alot of thing about connecting between core component(nova,cinder,...) to rabbit

[Boris Lukashev (rageltman)]

@VO LE HUY: yup, all sorts of services - designate, nova, cinder. Verified that they have proper authentication set and see them logging in as `openstack` successfully, immediately followed by the crash:
```
2023-06-22 05:12:31.897517+00:00 [info] <0.84962.0> accepting AMQP connection <0.84962.0> (10.0.2.10:44912 -> 10.0.2.12:5672)
2023-06-22 05:12:31.898704+00:00 [info] <0.84962.0> Connection <0.84962.0> (10.0.2.10:44912 -> 10.0.2.12:5672) has a client-provided name: designate-central:29677:f5298188-4683-45ba-b265-733af7c30123
2023-06-22 05:12:31.899375+00:00 [info] <0.84962.0> connection <0.84962.0> (10.0.2.10:44912 -> 10.0.2.12:5672 - designate-central:29677:f5298188-4683-45ba-b265-733af7c30123): user 'openstack' authenticated and granted access to vhost '/'
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> crasher:
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> initial call: rabbit_reader:init/3
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> pid: <0.84962.0>
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> registered_name: []
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> exception exit: {unexpected_message,{'EXIT',#Port<0.8188>,einval}}
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> in function rabbit_reader:handle_other/2 (rabbit_reader.erl, line 641)
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> in call from rabbit_reader:mainloop/4 (rabbit_reader.erl, line 532)
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> in call from rabbit_reader:run/1 (rabbit_reader.erl, line 454)
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> in call from rabbit_reader:start_connection/4 (rabbit_reader.erl, line 353)
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> ancestors: [<0.84960.0>,<0.53806.0>,<0.53805.0>,<0.53804.0>,<0.53802.0>,
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> <0.53801.0>,rabbit_sup,<0.53325.0>]
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> message_queue_len: 0
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> messages: []
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> links: [<0.84960.0>]
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> dictionary: [{process_name,
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> {rabbit_reader,
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> <<"10.0.2.10:44912 -> 10.0.2.12:5672 - designate-central:29677:f5298188-4683-45ba-b265-733af7c30123">>}},
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> {connection_user_provided_name,
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> <<"designate-central:29677:f5298188-4683-45ba-b265-733af7c30123">>},
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> {client_properties,
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> [{<<"product">>,longstr,<<"py-amqp">>},
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> {<<"product_version">>,longstr,<<"5.0.9">>},
2023-06-22 05:12:31.900039+00:00 [error] <0.84962.0> {<<"capa...

[Boris Lukashev (rageltman)]

Confirm that it _is_ in fact the container image and not the Ansible code.
Forcing use of the dockerhub image via `rabbitmq_image: "kolla/ubuntu-source-rabbitmq"` and then running the upgrade for the rabbitmq tag produces clean logs.

Had something simialar going on with the mariadb containers and had to drop STACKLEAK (of all things) to get them to replicate correctly - those were just failing silently.

[VO LE HUY (huyvl3)]

@Boris Lukashev (rageltman),

Do you mean all of image will pull from quay.io/openstack.kolla except rabbitmq from dockerhub? Its great solution. Can you show me file global.yaml just about "rabbit" image?

[Yusuf Güngör (yusuf2)]

Hi, we have the same problem when upgrading to xena.

In our case, it seems that xena rabbitmq image erlang version is upgraded to 26 and rabbitmq version is 3.9.29

According to rabbitmq documentation Maximum supported Erlang/OTP version is 25.2 for rabbitmq version 3.9.29 (https://www.rabbitmq.com/which-erlang.html)

rabbitmq-xena image id: 428f5f6c8fdd (created 30 hours ago)

It said that Erlan 26 is not officially supported for same problem: https://stackoverflow.com/questions/76274566/rabbitmq-closed-connection-on-method-createmodel

()[rabbitmq@7b64a8613ac1 /]$ erl --version
Erlang/OTP 26 [erts-14.0.1] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [jit:ns]

()[rabbitmq@7b64a8613ac1 /]$ rabbitmqctl --version
3.9.29

Probably same problem exist for yoga images too. See the bug : https://bugs.launchpad.net/kolla/+bug/2024888

Rabbitmq Logs:

2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> crasher:
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> initial call: rabbit_reader:init/3
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> pid: <0.503397.0>
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> registered_name: []
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> exception exit: {unexpected_message,{'EXIT',#Port<0.40446>,einval}}
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> in function rabbit_reader:handle_other/2 (rabbit_reader.erl, line 641)
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> in call from rabbit_reader:mainloop/4 (rabbit_reader.erl, line 532)
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> in call from rabbit_reader:run/1 (rabbit_reader.erl, line 454)
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> in call from rabbit_reader:start_connection/4 (rabbit_reader.erl, line 353)
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> ancestors: [<0.503395.0>,<0.950.0>,<0.949.0>,<0.948.0>,<0.946.0>,
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> <0.945.0>,rabbit_sup,<0.234.0>]
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> message_queue_len: 0
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> messages: []
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> links: [<0.503395.0>]
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> dictionary: [{connection_user_provided_name,
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> <<"neutron-server:8094:6075b4f3-6074-4ede-b819-a33041ced58b">>},
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> {{ch_pid,<0.503406.0>},
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> {1,#Ref<0.2514149007.2943352834.15353>}},
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> {client_properties,
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> [{<<"product">>,longstr,<<"py-amqp">>},
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> {<<"product_version">>,longstr,<<"5.0.8">>},
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> {<<"capabilities">>,table,
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> [{<<"authentication_failure_close">>,bool,true},
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> {<<"connection.blocked">>,bool,true},
2023-06-23 12:39:05.575444+03:00 [error] <0.503397.0> {<<...

[Boris Lukashev (rageltman)]

@Kolla team - how are these being QAd before being pushed to Quay? I'm guessing you're not running Tempest across the board, but a stack stand-up on a set of built container products to perform basic integration tests might not be a bad idea as it would catch this sort of failure before publishing the container.

@VO LE HUY: that single line of YAML is all you need in your variables definition, though not sure how that plays with the "binary" deployment type.

[Sergio Durigan Junior (sergiodj)]

Hello and thank you for taking the time to file a bug.

If I understand correctly, you are using a version of ansible-core that is not present in Focal. Also, we do not seem to have kolla-ansible in our official repositories. Therefore, it seems to me that you're using unsupported packages to do this deployment.

If you believe this is a bug in the rabbitmq-server shipped in Ubuntu, then please provide the steps to reproduce the problem *without* involving third-party, unsupported packages. This way, we can work on finding a proper fix for the package we officially ship.

I will mark this bug as Incomplete in order to reflect the fact that we are waiting for the requested steps to reproduce the bug. Please feel free to set the bug status back to New once you provide this info.

Thanks.

[VO LE HUY (huyvl3)]

@Sergio Durigan Junior (sergiodj),

My ubuntu-20.04 installed ansible-core:
$ apt-get update -y
$ sudo apt-get install software-properties-common -y
$ sudo add-apt-repository --yes --update ppa:ansible/ansible
$ sudo apt-get install ansible -y

Was I wrong?

[Nikita Lapenkov (nlapenkov)]

Hello!

I deployed openstack via this manual
https://docs.openstack.org/kolla-ansible/xena/user/quickstart.html#install-kolla-ansible

and have crash rabbit too

[Sven Kieske (s-kieske)]

imho this is better addressed in kolla-ansible, as it seems to be specific to a kolla-ansible deployed rabbitmq

[Sven Kieske (s-kieske)]

where did you get this old rabbitmq version from?

rmq 3.9 is out of support upstream: https://www.rabbitmq.com/versions.html

I'm 90% sure we don't reference this old version anymore but want to doublecheck with you.

it would also be good to not use binary builds, but use source builds of our containers, as the binary containers are not really supported anymore.

[Boris Lukashev (rageltman)]

@Sven Kieske: please check the images in Quay for Yoga on 20.04, even when using the "source" option this problem causes upgrades to fail (i dont use the binary one). Given the workaround i posted, this is something which has crept into the images over the last few months since the ones on dockerhub still work.

[Yusuf Güngör (yusuf2)]

+1 for @rageltman

[Yusuf Güngör (yusuf2)]

Xena latest commit: rabbitmq: update to RabbitMQ 3.9 on Erlang 25
 (https://github.com/openstack/kolla/commit/f3edd4d6706c97da9c954edcac6ef976366a073e)

But installed erlan version is 26

()[rabbitmq@a5cf2e3090ac /]$ apt list --installed|grep -i erlan

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

erlang-asn1/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-base/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-crypto/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-eldap/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-ftp/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-inets/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-mnesia/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-os-mon/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-parsetools/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-public-key/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-runtime-tools/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-snmp/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-ssl/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-syntax-tools/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-tftp/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-tools/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
erlang-xmerl/now 1:26.0.1-1rmq1ppa1~ubuntu20.04.1 amd64 [installed,local]
()[rabbitmq@a5cf2e3090ac /]$

[Sven Kieske (s-kieske)]

@yusuf2: from which ppa is that erlang?

can you post the output of:
cat /etc/apt/sources.list /etc/apt/sources.list.d/*

inside your rabbitmq container?

Thanks

[Yusuf Güngör (yusuf2)]

Hi Sven, thanks for your support. Output as follows:

()[rabbitmq@a5cf2e3090ac /]$ cat /etc/apt/sources.list /etc/apt/sources.list.d/*
# For non-x86 architectures we use sources.list.ubuntu.<arch>

# Default repos
deb mirror://mirrors.ubuntu.com/mirrors.txt focal main universe
deb mirror://mirrors.ubuntu.com/mirrors.txt focal-updates main universe
deb mirror://mirrors.ubuntu.com/mirrors.txt focal-security main universe

# Backports have a lower priority and must be explicitly installed to be used
deb http://archive.ubuntu.com/ubuntu/ focal-backports main universe

# We need to add the repo for the updated packages they provide. The main ones
# are qemu, libvirt, and openvswitch.
deb http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/xena main

# NOTE(hrw): repositories below are added into image when they are needed as
# separate files in /etc/apt/sources.list.d/ directory. For that purpose they
# are defined in kolla/repos.yaml file.
#
# Here they stay in case someone needs them.

# Elasticsearch, Logstash & Kibana repo
#deb https://artifacts.elastic.co/packages/5.x/apt stable main

# InfluxDB repo
#deb https://repos.influxdata.com/ubuntu focal stable

# Grafana repo
#deb https://packages.grafana.com/oss/deb stable main

# Fluentd repo
#deb http://packages.treasuredata.com/4/ubuntu/focal/ focal contrib

# Qdrouterd repo
#deb http://ppa.launchpad.net/qpid/released/ubuntu/ focal main

# rabbitmq repo
#deb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ focal main

# erlang repo
#deb http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu focal main

# MariaDB repo
#deb http://downloads.mariadb.com/MariaDB/mariadb-10.3/repo/ubuntu focal main
Uris: https://ppa.launchpadcontent.net/rabbitmq/rabbitmq-erlang/ubuntu
Components: main
Types: deb
Suites: focal
Signed-By: /etc/kolla/apt-keys/erlang-ppa.gpg
# used for adding custom repos by 'apt_sources_list' option of kolla-build.conf
Uris: https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu
Components: main
Types: deb
Suites: focal
Signed-By: /etc/kolla/apt-keys/rabbitmq.gpg
()[rabbitmq@a5cf2e3090ac /]$

[Yusuf Güngör (yusuf2)]

There is no problem for zed rabbitmq image from quay.io

$ docker run -it quay.io/openstack.kolla/rabbitmq:zed-ubuntu-jammy bash
()[rabbitmq@c233699c01cf /]$ rabbitmqctl --version
3.10.24
()[rabbitmq@c233699c01cf /]$ erl --version
Erlang/OTP 25 [erts-13.2.2.2] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:1] [jit:ns]

()[rabbitmq@14e8e3a1915b /]$ cat /etc/apt/sources.list /etc/apt/sources.list.d/*
# For non-x86 architectures we use sources.list.ubuntu.<arch>

# Default repos
deb mirror://mirrors.ubuntu.com/mirrors.txt jammy main universe
deb mirror://mirrors.ubuntu.com/mirrors.txt jammy-updates main universe
deb mirror://mirrors.ubuntu.com/mirrors.txt jammy-security main universe

# Backports have a lower priority and must be explicitly installed to be used
deb http://archive.ubuntu.com/ubuntu/ jammy-backports main universe

# We need to add the repo for the updated packages they provide. The main ones
# are qemu, libvirt, and openvswitch.
deb http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-updates/zed main

# NOTE(hrw): extra repositories are added into image when they are needed as
# separate files in /etc/apt/sources.list.d/ directory. For that purpose they
# are defined in kolla/template/repos.yaml file.
Uris: https://ppa.launchpadcontent.net/rabbitmq/rabbitmq-erlang-25/ubuntu
Components: main
Types: deb
Suites: jammy
Signed-By: /etc/kolla/apt-keys/erlang-ppa.gpg
# used for adding custom repos by 'apt_sources_list' option of kolla-build.conf
Uris: https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu
Components: main
Types: deb
Suites: jammy
Signed-By: /etc/kolla/apt-keys/rabbitmq.gpg
()[rabbitmq@14e8e3a1915b /]$

[Marius Gulie (margul01)]

Same thing here,
Kolla-ansible : stable/yoga
RabbitMQ : 3.9.29

[Marius Gulie (margul01)]

In my case

(rabbitmq)[root@rn-ops10-ctr-01 /]# erl --version
Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

(rabbitmq)[root@rn-ops10-ctr-01 /]# rabbitmqctl --version
3.9.29

[Boris Lukashev (rageltman)]

Zed doesn't do 20.04, so it's not broken. Guessing yoga on 22.04 is fine too

[Alejandro Garcia (agarciaws)]

I have successfully deployed Yoga by cherrypicking the fix that was proposed to master and Zed (https://review.opendev.org/c/openstack/kolla/+/886574) which pins the Erlang PPA to erlang-25.

I can confirm building the RabbitMQ Kolla image with this change installs rabbitm 3.9.29 with Erlang 25:

(rabbitmq)[root@whitecloud-node-0 /]# rabbitmqctl --version
3.9.29
(rabbitmq)[root@whitecloud-node-0 /]# erl --version
Erlang/OTP 25 [erts-13.2.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

I have just submitted the fix to the stable/yoga branch here:

https://review.opendev.org/c/openstack/kolla/+/887642

[VO LE HUY (huyvl3)]

+1 Alejandro Garcia (agarciaws)

[VO LE HUY (huyvl3)]

@Boris Lukashev (rageltman),

You can deploy Zed on Ubunt 20.04, its really simple. This is what I did

root@huyvl-test:~# history 10
    1 apt-get update -y && apt-get install vim git -y
    2 python3
    3 python3 -m pip install ansible
    4 apt-get install python3-pip -y
    5 python3 -m pip install ansible
    6 ansible --version
    7 pip3 install -U pip
    8 pip3 install git+https://opendev.org/openstack/kolla-ansible@stable/zed
    9 kolla-ansible --version

[Boris Lukashev (rageltman)]

@VO LE HUY: you're running the ansible commands on 20.04, kolla-ansible only supports 22.04 for Zed deployment targets - https://docs.openstack.org/kolla-ansible/zed/user/support-matrix.html.

[VO LE HUY (huyvl3)]

@Boris Lukashev (rageltman): after that installation you can define zed on global.yaml, every bootstrap/precheck/deploy will be successful, created VM and work very well. That what I tried without any error.

[Yusuf Güngör (yusuf2)]

Hi, is it possible to prepare a fix for stable/xena branch too? Thanks

[Alejandro Garcia (agarciaws)]

Yusuf, after merging my patch in stable/yoga we can cherry-pick it back to Xena and Wallaby.

Can you, Boris and VO LE HUY take a loook at it and add a review? https://review.opendev.org/c/openstack/kolla/+/887642

[VO LE HUY (huyvl3)]

Really thank, I see.

[VO LE HUY (huyvl3)]

Seem like they haven't merge yet. Take a longtime to this bug.

[Yusuf Güngör (yusuf2)]

Hi @agarciaws thanks you for your effort on this bug. I am not familiar with kolla-ansible code, not able to review it :(

[VO LE HUY (huyvl3)]

I waited too long for the patch. Don't know why @Zuul didn't merge?

[VO LE HUY (huyvl3)]

@Alejandro Garcia (agarciaws),

How can I update fix patch to my local environement? I'm using kolla-ansible, What file do I need to edit and where is it located?

[Sven Kieske (s-kieske)]

this should be fixed by https://review.opendev.org/c/openstack/kolla/+/886674.

don't know why this issue didn't autoclose.

[VO LE HUY (huyvl3)]

Thank so much.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla xena-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 14.11.0 release.