RabbitMQ role does not handle feature flag switching during upgrades

[Andrew Bonney]

RabbitMQ's feature flags setup requires that any listed in the 'Required' column for a specific version are enabled before upgrading to that version (https://www.rabbitmq.com/feature-flags.html#list-of-feature-flags). If the flags are not enabled before upgrading, the service will fail to start, for example:

Oct 09 10:52:55 ab-rmq-test rabbitmq-server[7262]: 2023-10-09 10:52:55.710535+00:00 [error] <0.232.0> BOOT FAILED
Oct 09 10:52:55 ab-rmq-test rabbitmq-server[7262]: 2023-10-09 10:52:55.710535+00:00 [error] <0.232.0> ===========
Oct 09 10:52:55 ab-rmq-test rabbitmq-server[7262]: 2023-10-09 10:52:55.710535+00:00 [error] <0.232.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}

This is only an issue for long-lived RabbitMQ installations, as feature flags appear to be enabled by default whenever a fresh install is performed using the first version which supports them. As such, some users may not encounter this if they upgrade their operating system and install RabbitMQ from scratch on a regular basis. In our case we encountered this with the 'maintenance_mode_status' flag, which suggests we originally installed a version before v3.8.8, and attempted to upgrade beyond v3.11 in 2023.1.

Note that feature flags can't be disabled once they are enabled. The 'Stable' description suggests that best practice is to enable these feature flags soon after you have upgraded to the version where they have become stable.

[Dmitriy Rabotyagov (noonedeadpunk)]

Very important part here, is that feature flags must be enabled BEFORE the upgrade. Moreover, some flags are becoming required since 3.12 for example according to the docs:

https://www.rabbitmq.com/feature-flags.html

So I assume it would be appropriate to enable all feature flags after upgrade is completed and backport that down couple of releases.

Of course, as an alternative we can enable flags before the upgrade to ensure that upgrade won't fail as some flag is now required.