upgrade 22.04 -> 24.04 won't start due to feature flags

[Hadmut Danisch]

[Impact]

Upgrading systems with rabbitmq-server installed will leave the server in a bad state because upgrading directly from the version in Jammy to the version in Noble is not supported by upstream rabbitmq-server.

[Test Plan]

For now, we add an upgrade quirk to prevent upgrades from Jammy to Noble if rabbitmq-server is installed, and provide an brief explanation to the user.

To test:

1. Create a Jammy container

$ lxc launch ubuntu:jammy jammy

2. Install rabbitmq-server

$ apt install -y rabbitmq-server

3. Attempt the upgrade

$ do-release-upgrade -d

The upgrade should abort with a message explaining why.

[Impact]

This quirk uses a common pattern in ubuntu-release-upgrader for similar purposes. If the package name was typo'd, it would not work correctly.

[Other information]

We may eventually need another SRU to revert this change if another solution is found.

[Original Description]

Hi,

I was just doing an upgrade from ubuntu 22.04 server to 24.04 server.

After upgrading, the rabbitmq-server cannot be started anymore.

Reason:

See
https://www.rabbitmq.com/blog/2022/07/20/required-feature-flags-in-rabbitmq-3.11

Higher versions of rabbitmq require feature flags to be set *before* upgrade. In my rabbitmq installation these features had not been set, therefore the the rabbitmq server would not start. They ask you to downgerade to an earlier version and do the upgrade.

I therefore used LXD to run a Ubuntu 22.04 machine with rabbitmq 3.9 to set the missing feature flags and achieved:

rabbitmqctl enable_feature_flag all

rabbitmqctl list_feature_flags
Listing feature flags ...
name state
implicit_default_bindings enabled
maintenance_mode_status enabled
quorum_queue enabled
stream_queue enabled
user_limits enabled
virtual_host_metadata enabled

But it still does not work, it still complains about one feature missing:

classic_mirrored_queue_version

unfortunately, this is required by rabbitmq 3.12 coming with Ubuntu 24.04, but is not known and thus cannot be set by rabbitmq 3.9 coming with Ubuntu 22.04.

Now the problem is: There is no Ubuntu coming with 3.9 or 3.10. Ubuntu jumps directly from rabbitmq 3.9 to rabbitmq 3.12, although there is no upgrade path from 3.9 to 3.12.

Three nasty options:

- loose your data and configuration and run from scratch
- run an old version in either docker or LXD
- try some upgrade path with docker/podman and non-ubuntu versions of rabbitmq

Unfortunately,

https://hub.docker.com/_/rabbitmq

does not list versions older than 3.12, but they still seem to be available, so this might by some migration/workaround path.

[Hadmut Danisch (hadmut)]

OK, found a workaround. From the 24.04 system, make sure to

systemctl stop rabbitmq-server.service
systemctl stop epmd.service

make a safety copy/backup of /var/lib/rabbitmq

Repeat the following two podman commands for 3.9 , 3.10, and 3.11:

In one shell run (replace 127 and 138 with uid and gid of rabbitmq on your system and make sure that $HOST is set to your hostname, must be the same as your host for things to work, the -p ports are not needed, but a way to make sure no other daemon is running)

podman run -it --rm -v /var/lib/rabbitmq:/var/lib/rabbitmq --uidmap=0:0 --uidmap=u999:127 --gidmap=0:0 --gidmap=g999:138 -p 5672:5672 -p 15672:15672 --name $HOST -h $HOST docker.io/library/rabbitmq:3.9

wait for it to come up and run from a second shell

podman exec -it -u rabbitmq $HOST rabbitmqctl enable_feature_flag all

(or run /bin/bash and check with
rabbitmqctl list_feature_flags
rabbitmqctl enable_feature_flag all
rabbitmqctl list_feature_flags
)

after doing this three times with 3.9 3.10 and 3.11

restart services epmd and then rabbitmq-server , and things should work now. Once happy, you can remove the podman images.

[Hadmut Danisch (hadmut)]

Forgot: Once finished with the second podman command, go back to the first shell, check the log about feature setting, and terminate with ctrl-c.

[Mitchell Dzurick (mitchdz)]

Thank you for reporting this bug and even following up on it Hadmut!

This sounds like something that could potentially be fixed in the Noble maintainer scripts.

I'd want to find the exact flags we need and enable just those, since you can't disable feature flags once they are enabled.

[Hadmut Danisch (hadmut)]

Problem 1: I have no idea how long docker-hub and the rabbitmq people will offer these outdated docker images for download. Maybe it is advisable to pull and save a copy.

Problem 2: This could (as it did with me, but I chose to update with the -d flag, so my own fault) drive people into severe trouble, if they do upgrade their production server or desktop from 22.04 to 24.04.1 and thus break rabbitmq-server and make their stored data unavailable. So it should be a stopper for do-release-upgrade and at least warn people before upgrading as long as this isn't fixed. 24.04.1 is announced to be released in two weeks, and .1 usually enables to upgrade LTS-to-LTS.

Problem 3: It is possible that this workaround does not smoothly run within LXD containers in current 24.04. Today I ran into the problem that after upgrading to 24.04 docker cannot be run within LXD containers anymore, because of some strange kernel/apparmor/runc problem, although there is a simple, but not obvious fix for that. Some people say that podman cannot be run either because of the same problem.

I was able fix all these problems, but took me some time, and it might overstrain average users.

regards
Hadmut

[Mitchell Dzurick (mitchdz)]

Are you running rabbitmq-server on the host, or in some container?

I just ran a quick test in an LXD container doing do-release-upgrade and it seems to work fine to me after upgrading to Noble.

Were there any special configuration options you had enabled in Jammy before the upgrade?

[Mitchell Dzurick (mitchdz)]

To more explicitly say what I tested:

I ran the following commands to create a Jammy container and run the upgrade process:
lxc launch ubuntu:jammy j-n-upgrade
lxc shell j-n-upgrade
apt update -y
apt install -y rabbitmq-server
apt install -y ubuntu-release-upgrader-core
sed -i 's/Prompt=lts/Prompt=normal/' /etc/update-manager/release-upgrades
do-release-upgrade -f DistUpgradeViewNonInteractive

Then after the upgrade was done, I created a simple rabbitmq-server script which I attached.
# apt install -y python3-pika
# ./test.sh

[Hadmut Danisch (hadmut)]

I was running it directly on the host, but noticed the docker problem when checking, why other services (not rabbitmq) I'm running within an LXD container didn't start.

Your tests might not show the problem, because you've made a fresh install, and rabbitmq-server seems to automatically set flags when the database is newly created. What features had been set in your test environment?

My rabbitmq-installation was running for years and already went through several release upgrades, therefore, none of the feature flags had been set in my database. (Until yesterday, I did not even know that these feature flags exist.)

So testing with a freshly created database might not show the problem.

The best way to deal with the problem would be to put something in the do-release-upgrade script which checks if rabbitmq-server is installed. If not – do nothing, proceed.

But if so, issue a warning text and give the choice to abort or proceed.

In the text, explain, that there is no direct stable upgrade path from 3.9 to 3.12, and that it could be possible under certain circumstances, that the server won't start after upgrade. People should urgently be requested to make a backup of /var/lib/rabbitmq, and make sure to set the recommended flags (or do set it in the upgrade procedure as a third option), and pointed to instructions about what to do if the service does not start after upgrading, or proposed to accept a complete loss and recreate the database from scratch (what could be acceptable in many use cases).

BTW, I just saw that there already are closely related bugs about the same problem: Bug #2038818 and Bug #2046665

[Mitchell Dzurick (mitchdz)]

Jammy:
root@j:~# rabbitmqctl list_feature_flags
Listing feature flags ...
name state
implicit_default_bindings enabled
maintenance_mode_status enabled
quorum_queue enabled
stream_queue enabled
user_limits enabled
virtual_host_metadata enabled

Noble (after upgrading from Jammy):
root@j-n-upgrade:~# rabbitmqctl list_feature_flags
Listing feature flags ...
name state
classic_mirrored_queue_version enabled
classic_queue_type_delivery_support enabled
direct_exchange_routing_v2 enabled
feature_flags_v2 enabled
implicit_default_bindings enabled
listener_records_in_ets enabled
maintenance_mode_status enabled
quorum_queue enabled
restart_streams enabled
stream_queue enabled
stream_sac_coordinator_unblock_group enabled
stream_single_active_consumer enabled
tracking_records_in_ets enabled
user_limits enabled
virtual_host_metadata enabled

[Mitchell Dzurick (mitchdz)]

Thanks for posting the other bugs, I tracked 2046665 to rabbitmq-server.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

What openstack does (from LP: #2038818):

https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/919701

[Christian Ehrhardt  (paelzer)]

Thanks Andreas,
I furthermore checked with "our" openstack team if that isn't something they've already hit, but they have no jammy -> noble upgrade path for their use of rabbitmq yet. Although they offered help to verify any fix we might propose by test upgrading their rabbitmq node using it.

This flag handling is indeed is a common practice, as for example in 3.11 needed all flags of 3.8.x just the same way [1].

The bonus problem that might make this harder, is that [2] states that 3.12 needs all flags of the 3.11.x series. But Jammy to Noble is 3.9.x -> 3.12 - so we might need an interim 3.11?!? That makes this quite complex unless e.g. a backport of the lastest 3.9.x makes that possible too the outcome will be quite complex.
Yet on the other hand, while some text says "before upgrade" and I do not see that in the OpenStack solution [4] Andreas found - to be fair that is so abstracted that I'm even yet unsure what/when it exactly does.

I must admit that I think we need a bit more time to get a better understanding of this [3] in general.
And even then we might still want to avoid too many assumptions and just ask upstream if there is any 3.9->3.12 way without an interim 3.11 - and discuss a more detailed plan of action from there.

Either way, big thanks to Hadmut to bring this to our attention!

[1]: https://www.rabbitmq.com/blog/2022/07/20/required-feature-flags-in-rabbitmq-3.11
[2]: https://github.com/rabbitmq/rabbitmq-server/discussions/8456
[3]: https://www.rabbitmq.com/docs/feature-flags
[4]: https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/919701

[Mitchell Dzurick (mitchdz)]

There is a github issue mentioning this[0]. There is potential plans to improve feature flags[1], but they do not help us right now, and will not help users that encounter this situation.

It seems the properly supported upgrade path is indeed installing 3.10 -> 3.11.

[0] - https://github.com/rabbitmq/rabbitmq-server/discussions/11418
[1] - https://github.com/rabbitmq/rabbitmq-server/issues/9677

[Mitchell Dzurick (mitchdz)]

I made an upstream bug report directly asking for support doing these major version hops[0].

[0] - https://github.com/rabbitmq/rabbitmq-server/discussions/11938

[Andreas Hasenack (ahasenack)]

I added an ubuntu-release-upgrader task because we might consider blocking the release upgrade there if the conditions for this bug are found to be present.

Those conditions might be as simple as "is rabbitmq-server installed (and running)?". Or perhaps a more generic one, carefully checking the version number jump between jammy and noble, because in the future we might have a newer version in jammy exactly because of this bug.

[Mitchell Dzurick (mitchdz)]

I just want to update on my upgrade test.

First and foremost - a direct upgrade from Jammy to Noble is not possible.

It only is if you have it installed and have not used rabbitmq-server, thus no metadata is made yet.

If you use the server in Jammy and then attempt to upgrade to Noble right now you will see:

Errors were encountered while processing:
 rabbitmq-server
needrestart is being skipped since dpkg has failed
packages have been installed but needrestart is suspended
packages have been installed but needrestart is suspended
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)
Setting up rabbitmq-server (3.12.1-1ubuntu1) ...
Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xeu rabbitmq-server.service" for details.
invoke-rc.d: initscript rabbitmq-server, action "restart" failed.
● rabbitmq-server.service - RabbitMQ Messaging Server
     Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2024-08-09 18:08:11 UTC; 7ms ago
    Process: 34558 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE)
   Main PID: 34558 (code=exited, status=1/FAILURE)
     Status: "Standing by"
        CPU: 1.631s
invoke-rc.d: release upgrade in progress, error is not fatal
root@j-n-upgrade2:~# lsb_Release -a
Command 'lsb_Release' not found, did you mean:
  command 'lsb_release' from deb lsb-release (12.0-2)
Try: apt install <deb name>
root@j-n-upgrade2:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
root@j-n-upgrade2:~# systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ Messaging Server
     Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2024-08-09 18:12:26 UTC; 7s ago
    Process: 36160 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE)
   Main PID: 36160 (code=exited, status=1/FAILURE)
     Status: "Standing by"
        CPU: 1.566s

[Mitchell Dzurick (mitchdz)]

Hadmut, thanks again for bringing this to our attention.

With 24.04.1 coming out soon we may expect more upgrades from Jammy -> Noble. Therefore I am looking to block the upgrade (22.04 -> 24.04) if rabbitmq-server is installed.

This is unfortunate, but I think is a fair to not make rabbitmq-server unusable for users after upgrade. This will also give us time to evaluate a proper upgrade path that can be relied upon, and allow users to backup data before attempting anything.

Likely it will be a manual process of upgrading interim releases (rabbitmq-server 3.9.X -> 3.10.X -> 3.11.X), or accepting that you will need to recreate your server's database, but that's something I will be working on.

[Hadmut Danisch (hadmut)]

You're welcome. :-)

As I commented in #1, the upgrade using podman was working well for me, but the question is, how long these docker images of old versions will be available.

When creating an upgrade strategy, you also should take into consideration, how long this upgrade should work, i.e. whether the upgrade path should still work in 2027 when 22.04's support will end.

Another problem is that 24.04 is still stuck to rabbitmq 3.12, which is out of support and does not receive updates anymore, thus could be a security problem and require an emergency update 3.12 -> 3.13 if a security problem is found.

Maybe it would be easier and less complex, to first update 24.04 from rabbitmq 3.12 to 3.13, which seems to be the last 3.x version (they are beta-testing 4.x), as a reasonable version for 24.04 LTS, and check whether there's a similar upgrade problem 3.12->3.13, and incorporate this into your solution.

And, keep in mind, you probably will have similar problems with future versions. The same problem will probably reoccur.

Maybe, one clean and simple solution could be to decouple rabbitmq-packages from Ubuntu releases and simply turn them into snaps with a channel for each version, i.e. do no upgrade at all and just replace ubuntu package 3.9 with snap channel 3.9 during update to keep things running and benefit from snap's revert mechanism in case something doesn't start.

[Mitchell Dzurick (mitchdz)]

Great points Hadmut.

It is unfortunate how far behind we are in versions. We follow debian which hasn't been updated in a bit. I was thinking about updating the package there, but I need to fix the fires here first (and if I update debian to the latest without handling this bug's issue, I will break people's server).

I am definitely interested in updating rabbitmq-server to 3.13, however we do not often update major versions because they are prone to breaking things (hence, this bug report). If anything, I would consider making a new binary `rabbitmq-server-3.13` similar to how say, gcc-13 does it. That way users can upgrade to the new version but it's more of an informed decision rather than things just breaking upon an `apt update && apt upgrade`.

Even if we stay on top of it, users might not `apt update && apt upgrade` for an extended period of time, and then we would have to deal with that.

You are on point, I was also thinking about the potential of snaps long term.

[Hadmut Danisch (hadmut)]

There's another option, although an ugly one.

If it is too bulky to be supported in a clean way, and too incompatible with Ubuntu's upgrade cycles, and if the problem with upgrading can't be properly solved in the Ubuntu universe, it could simply be dropped from Ubuntu (and Debian), and users be redirected to either use rabbitmq's own apt repository, or docker/podman/kubernetes containers.

After all, hasn't Ubuntu just invented a more secure way to use third party apt repositories for 24.04?

[Łukasz Zemczak (sil2100)]

Hello Hadmut, or anyone else affected,

Accepted ubuntu-release-upgrader into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:24.04.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

I verified the upgrade quirk using ubuntu-release-upgrader from noble-proposed:

nr@six:~$ lxc launch ubuntu:jammy jammy
nr@six:~$ lxc exec jammy bash
root@jammy:~# apt install -y rabbitmq-server
[ ...SNIP... ]
root@jammy:~# do-release-upgrade --proposed
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@jammy:~# sed -i 's/Prompt=lts/Prompt=normal/g' /etc/update-manager/release-upgrades
root@jammy:~# do-release-upgrade --proposed
[ ...SNIP... ]

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done

Unable to upgrade to Ubuntu 24.04 LTS

Currently, you have RabbitMQ server installed, which is not directly
upgradable to the newer version. Upgrading may prevent the server
from starting due to missing feature flags.

For more information, please see
https://bugs.launchpad.net/bugs/2074309.

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done

[Łukasz Zemczak (sil2100)]

Hello Hadmut, or anyone else affected,

Accepted ubuntu-release-upgrader into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:24.04.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

The most recent upload contains a new quirk for an unrelated bug, and fixes for another unrelated bug. The previous verification done with 1:24.04.21 stands.

[Hadmut Danisch (hadmut)]

Hi,

actually I do not have any problem to fix anymore since the workaround procedure with podman described above totally solved my problem. (Improvement: replace 127 with $(id -u rabbitmq) and 138 with $(id -g rabbitmq)

But to test the bug fix I've setup a fresh 22.04 in LXD, installed rabbitmq-server, enabled -proposed, copied my backup of my old server into /var/lib/rabbitmq and ran a do-release-upgrade -p.

And yes, the upgrade was denied:

Unable to upgrade to Ubuntu 24.04 LTS

Currently, you have RabbitMQ server installed, which is not directly
upgradable to the newer version. Upgrading may prevent the server
from starting due to missing feature flags.

For more information, please see
https://bugs.launchpad.net/bugs/2074309.

Ursprünglicher Systemzustand wird wieder hergestellt

Wird abgebrochen

So yes, it kept me from breaking the server.

However, I'd suggest to extend the message a bit, since a not so experienced admin just wouldn't know what to do now.

I'd suggest to tell the three options:

- stay safe, but outdated at 22.04 (for unknown time)

- lose your database and all your data, and complete purge the rabbitmq-server package, do update, and do fresh install from scratch

- stop daemon, backup /var/lib/rabbitmq, remove, but not purge the package, do the upgrade, use the workaround procedure with podman as described above (and no warranty that it works and images are still available), and reinstall/start.

As far as I know, Ubuntu has a Wiki. Wouldn't it be better to make a wiki page with instructions instead of pointing to this bug?

[Mitchell Dzurick (mitchdz)]

Hi Hadmut, yes I agree. Unfortunately it's a bit easier to modify this bug than the ubuntu release ugprade message, hence why the message in the upgrader is so sparse. The hopeful plan is to have a nice wiki page to redirect to.

I'm working on a good alternative upgrade path that we can rely on for the future.

Personally, I'd rather have all the information in this bug rather than copying information in multiple places, so we have one source of truth to follow.

P.S. I really appreciate all your effort here Hadmut.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:24.04.22

---------------
ubuntu-release-upgrader (1:24.04.22) noble; urgency=medium

  [ Tobias Heider ]
  * Fix typo in force_obsoletes. (LP: #2076913)

  [ Nick Rosbrook ]
  * DistUpgrade: handle cache key error when removing obsolete packages
    (LP: #2077358)
  * deb2snap: switch snap-store and firmware-updater tracks
    (LP: #2077344)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

ubuntu-release-upgrader (1:24.04.21) noble; urgency=medium

  * DistUpgradeQuirks: disable stub resolver on upgrades from jammy
    (LP: #2043820)
  * DistUpgrade: show a message about denied package removals
    (LP: #2073278)
  * data: add cryptsetup-initramfs to removal_denylist.cfg
    (LP: #2067622)
  * DistUpgradeQuirks: abort if system is not /usr-merged
    (LP: #2058648)
  * DistUpgradeQuirks: disable cloud-init on upgrade
    (LP: #2075968)
  * DistUpgradeQuirks: abort if rabbitmq-server installed
    (LP: #2074309)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

ubuntu-release-upgrader (1:24.04.20) noble; urgency=medium

  [ Mate Kukri ]
  * DistUpgradeQuirks: make sure GRUB "cloud_style_installation" is set in cloud images (LP: #2054103)

  [ Julian Andres Klode ]
  * Move the actiongroup out of the obsoletes loop
  * Reimplement obsolete removal with resolve_by_keep (LP: #1874272)

  [ Nick Rosbrook ]
  * Run pre-build.sh: updating mirrors, demotions, and translations.

 -- Nick Rosbrook <email address hidden> Wed, 21 Aug 2024 12:55:15 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Olsen (jolsenmitsu)]

Not sure if this should be a new report or added to this one, so I'll start here and create a new report if directed to.

I am trying to execute do-release-upgrade and I'm getting this error message:

  Currently, you have RabbitMQ server installed, which is not directly
  upgradable to the newer version. Upgrading may prevent the server
  from starting due to missing feature flags.

I am currently on Ubuntu 22.04.5 LTS and have executed apt update and upgrade.

My RabbitMQ seems to be the latest RabbitMQ release, according to rabbitmq.com
  sudo rabbitmqctl version
returns:
  3.13.7

All feature flags are currently enabled except for khepri_db:
  Listing feature flags ...
  name state
  classic_mirrored_queue_version enabled
  classic_queue_type_delivery_support enabled
  detailed_queues_endpoint enabled
  direct_exchange_routing_v2 enabled
  drop_unroutable_metric enabled
  empty_basic_get_metric enabled
  feature_flags_v2 enabled
  implicit_default_bindings enabled
  khepri_db disabled
  listener_records_in_ets enabled
  maintenance_mode_status enabled
  message_containers enabled
  message_containers_deaths_v2 enabled
  quorum_queue enabled
  quorum_queue_non_voters enabled
  restart_streams enabled
  stream_filtering enabled
  stream_queue enabled
  stream_sac_coordinator_unblock_group enabled
  stream_single_active_consumer enabled
  stream_update_config_command enabled
  tracking_records_in_ets enabled
  user_limits enabled
  virtual_host_metadata enabled

I'm not sure what exactly the issue is, especially since it seems like I'm already on the latest version of RabbitMQ. My assumption (like wrong) is there is an issue with one or more tests in the ubuntu-release-upgrader.

[Melroy van den Berg (melroyvandenberg)]

I am also affected. And was stopped nicely by the do-release-upgrade command. Hopefully RabbitMQ can soon fix this issue, so I can upgrade my Ubuntu Server to 24.04.

[Mitchell Dzurick (mitchdz)]

Hi James,

I apologize, we made the check if rabbitmq-server is installed no matter the version. If you are using the upstream PPA and Erlang version this bug shouldn’t apply to you. Let me see if I can get that check updated.

[Mitchell Dzurick (mitchdz)]

Melroy, thanks for taking the time to comment.

Currently the issue is that the feature flags don’t get updated when doing the major version hops. These feature flags are part of your mnesia database, so if your cluster isn’t complicated you could uninstall rabbit-server, dust-upgrade, install rabbitmq-server, and then recreate the cluster. It’s a bit manual right now, but it’s the best option that’s available right now, but I’m looking into better options moving forward (including ones similar to the podman route).

[kelv (yskchu)]

Hi Team,

I am also using the upstream PPA and Erlang version per RabbitMQ instructions (source list info below), and I'm already on version 4.0.2. I am not affected by any feature flag issues for the 22.04 to 24.04 upgrade.

Same as others, I can't upgrade my server because of the way the do-release-upgrade check is currently done.

Is there an override for do-release-upgrade so we can apply so we can upgrade our servers from 22.04 to 24.04?

Alternatively, the easiest way is for your do-release-upgrade to check if the current installed rabbitmq version is already at 3.12 or higher - if so there is no impact to the upgrade.

Thanks!
Kelv

Source list info below (I will change them to noble after upgrade):

## Provides modern Erlang/OTP releases
##
deb [signed-by=/my/key/location/erlang.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/my/key/location/erlang.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main

## Provides RabbitMQ
##
deb [signed-by=/my/key/location/rabbit.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/my/key/location/rabbit.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu jammy main

[Melroy van den Berg (melroyvandenberg)]

@kelv No, I don't think the do-release-upgrade command allows you to by-pass this check. So try what Mitchell just said above your comment, try to remove rabbitmq-server instead. Try proceed with the Ubuntu upgrade using do-release-upgrade CLI command. And finally try to install RabbitMQ & recreate the cluster.

Since my cluster is also not very complicated and fully managed by the Symfony framework. I might also give this a try.

[Hadmut Danisch (hadmut)]

@kelv:

- Backup your database dir

- remove the rabbitmq-packages

- do the upgrade

- reinstall the packages

- If you didn't make a mistake, the databasedir should still be in place and work. If not, stop the server, restore the backup, and restart.

pretty simple.

[kelv (yskchu)]

@hadmut

I disagree with your "pretty simple" statement.

So the Ubuntu stance is that a poorly coded do-release-upgrade check should delay upgrades for all unaffected users, while the suggested workaround is to ask users to uninstall packages, rather than correcting the flawed do-release-upgrade check?

Uninstalling packages can lead to unforeseen issues and must be thoroughly tested to avoid potential complications. This approach seems impractical and may cause additional problems for users. Could a more efficient solution be implemented to address the underlying issue in the do-release-upgrade check itself?

Moreover, the current do-release-upgrade notice doesn't even mention the uninstalling-reinstalling process. Since updating the notice requires modifying the do-release-upgrade check anyway, why not take this opportunity to fix the check properly?

[Mitchell Dzurick (mitchdz)]

I’m thinking to make the u-r-u check a warning during upgrade telling the user about this bug, and allowing them to upgrade with a y/n prompt. This will allow you to make an informed decision before upgrading. Please give us time to make this change.

[Andreas Hasenack (ahasenack)]

I filed a separate bug to track this improvement in the release upgrader:

https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/2082004

Please subscribe that one to be made aware of the coming changes.

[kelv (yskchu)]

@michdz and @ahasenack thank you very much.

Separately, perhaps this is the right place to discuss the plan for RabbitMQ packages in Ubuntu moving forward. This will likely be an issue again in 1.5 years when the next LTS drops.

As mentioned by the RabbitMQ team in [this discussion raised by michdz earlier](https://github.com/rabbitmq/rabbitmq-server/discussions/11938), the release cadence of RabbitMQ currently does not align with Ubuntu LTS support timelines. To avoid similar issues in the future, it might be necessary to include these intermediate RabbitMQ releases so that users can upgrade seamlessly, or alternatively, direct users to the RabbitMQ managed repository.

While I applaud the ingenuity of the Podman workaround suggested by some members, users need to test feature flags first (this is why RabbitMQ itself does not enable feature flags automatically). Given that testing is necessary, it seems the only viable solution is to provide users with the intermediate packages, enabling them to upgrade incrementally.

[Mitchell Dzurick (mitchdz)]

You are completely right kelv.

In addition to the points you added, a big issue here is that Erlang currently lags in Debian. We would have to supply an updated host Erlang which could deviate from Debian and cause issues down the road.

I'm not a big fan of this situation, I'd rather work with upstream than patching EOL versions.