upgrade 22.04 -> 24.04 won't start due to feature flags
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Release Notes for Ubuntu |
New
|
Undecided
|
Unassigned | |||
rabbitmq-server (Ubuntu) | Status tracked in Oracular | |||||
Noble |
Confirmed
|
High
|
Unassigned | |||
Oracular |
Confirmed
|
High
|
Unassigned | |||
ubuntu-release-upgrader (Ubuntu) | Status tracked in Oracular | |||||
Noble |
Fix Committed
|
Undecided
|
Unassigned | |||
Oracular |
New
|
Undecided
|
Unassigned |
Bug Description
[Impact]
Upgrading systems with rabbitmq-server installed will leave the server in a bad state because upgrading directly from the version in Jammy to the version in Noble is not supported by upstream rabbitmq-server.
[Test Plan]
For now, we add an upgrade quirk to prevent upgrades from Jammy to Noble if rabbitmq-server is installed, and provide an brief explanation to the user.
To test:
1. Create a Jammy container
$ lxc launch ubuntu:jammy jammy
2. Install rabbitmq-server
$ apt install -y rabbitmq-server
3. Attempt the upgrade
$ do-release-upgrade -d
The upgrade should abort with a message explaining why.
[Impact]
This quirk uses a common pattern in ubuntu-
[Other information]
We may eventually need another SRU to revert this change if another solution is found.
[Original Description]
Hi,
I was just doing an upgrade from ubuntu 22.04 server to 24.04 server.
After upgrading, the rabbitmq-server cannot be started anymore.
Reason:
See
https:/
Higher versions of rabbitmq require feature flags to be set *before* upgrade. In my rabbitmq installation these features had not been set, therefore the the rabbitmq server would not start. They ask you to downgerade to an earlier version and do the upgrade.
I therefore used LXD to run a Ubuntu 22.04 machine with rabbitmq 3.9 to set the missing feature flags and achieved:
rabbitmqctl enable_feature_flag all
rabbitmqctl list_feature_flags
Listing feature flags ...
name state
implicit_
maintenance_
quorum_queue enabled
stream_queue enabled
user_limits enabled
virtual_
But it still does not work, it still complains about one feature missing:
classic_
unfortunately, this is required by rabbitmq 3.12 coming with Ubuntu 24.04, but is not known and thus cannot be set by rabbitmq 3.9 coming with Ubuntu 22.04.
Now the problem is: There is no Ubuntu coming with 3.9 or 3.10. Ubuntu jumps directly from rabbitmq 3.9 to rabbitmq 3.12, although there is no upgrade path from 3.9 to 3.12.
Three nasty options:
- loose your data and configuration and run from scratch
- run an old version in either docker or LXD
- try some upgrade path with docker/podman and non-ubuntu versions of rabbitmq
Unfortunately,
https:/
does not list versions older than 3.12, but they still seem to be available, so this might by some migration/
tags: | added: server-triage-discuss |
Changed in rabbitmq-server (Ubuntu): | |
importance: | Undecided → High |
Changed in rabbitmq-server (Ubuntu Noble): | |
milestone: | none → ubuntu-24.04.1 |
status: | New → Confirmed |
importance: | Undecided → High |
summary: |
- rabbitmq-server upgrade 22.04 -> 24.04 completely broken + rabbitmq-server upgrade 22.04 -> 24.04 won't start due to feature flags |
summary: |
- rabbitmq-server upgrade 22.04 -> 24.04 won't start due to feature flags + upgrade 22.04 -> 24.04 won't start due to feature flags |
Changed in ubuntu-release-upgrader (Ubuntu Noble): | |
status: | New → In Progress |
description: | updated |
OK, found a workaround. From the 24.04 system, make sure to
systemctl stop rabbitmq- server. service
systemctl stop epmd.service
make a safety copy/backup of /var/lib/rabbitmq
Repeat the following two podman commands for 3.9 , 3.10, and 3.11:
In one shell run (replace 127 and 138 with uid and gid of rabbitmq on your system and make sure that $HOST is set to your hostname, must be the same as your host for things to work, the -p ports are not needed, but a way to make sure no other daemon is running)
podman run -it --rm -v /var/lib/ rabbitmq: /var/lib/ rabbitmq --uidmap=0:0 --uidmap=u999:127 --gidmap=0:0 --gidmap=g999:138 -p 5672:5672 -p 15672:15672 --name $HOST -h $HOST docker. io/library/ rabbitmq: 3.9
wait for it to come up and run from a second shell
podman exec -it -u rabbitmq $HOST rabbitmqctl enable_feature_flag all
(or run /bin/bash and check with
rabbitmqctl list_feature_flags
rabbitmqctl enable_feature_flag all
rabbitmqctl list_feature_flags
)
after doing this three times with 3.9 3.10 and 3.11
restart services epmd and then rabbitmq-server , and things should work now. Once happy, you can remove the podman images.