SLURP upgrade from 2023.1 to master (2024.1) fails with: restart_streams feature flag is not available

Bug #2058512 reported by Will Szumski
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-ansible
In Progress
Undecided
Unassigned

Bug Description

The error I am seeing is:

TASK [rabbitmq : Enable feature flags] *****************************************
Wednesday 20 March 2024 13:33:14 +0000 (0:00:00.099) 0:00:21.630 *******
ok: [controller0] => (item=quorum_queue)
ok: [controller0] => (item=implicit_default_bindings)
ok: [controller0] => (item=virtual_host_metadata)
ok: [controller0] => (item=maintenance_mode_status)
ok: [controller0] => (item=user_limits)
ok: [controller0] => (item=stream_queue)
ok: [controller0] => (item=classic_queue_type_delivery_support)
ok: [controller0] => (item=tracking_records_in_ets)
ok: [controller0] => (item=stream_single_active_consumer)
ok: [controller0] => (item=listener_records_in_ets)
ok: [controller0] => (item=feature_flags_v2)
ok: [controller0] => (item=direct_exchange_routing_v2)
ok: [controller0] => (item=classic_mirrored_queue_version)
ok: [controller0] => (item=stream_sac_coordinator_unblock_group)
failed: [controller0] (item=restart_streams) => {"action": "community.rabbitmq.rabbitmq_feature_flag", "ansible_loop_var": "item", "changed": false, "item": "restart_streams", "msg": "restart_streams feature flag is not available"}

This is when attempting to upgrade rabbitmq. We seem to have 3.11.28 in 2023.1, but restart_streams wasn't added until 3.12:

https://www.rabbitmq.com/docs/feature-flags#core-feature-flags

It seems like in the upgrade job we do a reconfigure prior to the upgrade (which essentially gives us the master containers). This is not a documented step.

Will Szumski (willjs)
summary: - SLURP upgrade from 2023.1 to 2024.1 fails with: restart_streams feature
- flag is not available
+ SLURP upgrade from 2023.1 to master (2024.1) fails with: restart_streams
+ feature flag is not available
description: updated
Revision history for this message
Matt Crees (mattcrees) wrote :

The reconfigure step was added when we were using HA queues. Given we are now instead enabling quorum queues by default, we do not need this reconfigure step anymore. See docs here for the required steps: https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)
Changed in kolla-ansible:
status: New → In Progress
Revision history for this message
Michal Nasiadka (mnasiadka) wrote :

Seems we should rework the feature flags to check for rabbitmq version when applying them

Revision history for this message
Michal Nasiadka (mnasiadka) wrote :

Or just drop the reconfigure ;-)

Revision history for this message
Will Szumski (willjs) wrote (last edit ):

You can enabled them all like this:

    rabbitmqctl enable_feature_flag all

https://www.rabbitmq.com/docs/upgrade#enable-required-feature-flags-before-attempting-the-upgrade

But of course the ansible module doesn't seem to support that.

Edit: Tried to add support here: https://github.com/ansible-collections/community.rabbitmq/pull/170

Revision history for this message
Will Szumski (willjs) wrote :

Just to note that this has been discussed in the weekly meetings and the solution was to backport the newer rabbit back to 2023.1 so we can do the upgrade on 2023.1.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/914535
Committed: https://opendev.org/openstack/kolla-ansible/commit/223556e4e2efe35c77c4607408d47e67577a68f1
Submitter: "Zuul (22348)"
Branch: master

commit 223556e4e2efe35c77c4607408d47e67577a68f1
Author: Matt Crees <email address hidden>
Date: Wed Mar 27 16:15:54 2024 +0000

    RMQ: enable all stable feature flags at once

    Also enable these after an upgrade.

    Partial-Bug: #2058512
    Change-Id: Ib9bdae2e25c2b6cce30e4c8024015ab5875bc1ff

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/922606

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/922607

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/922607
Committed: https://opendev.org/openstack/kolla-ansible/commit/2b4165c106d05b6ef760b8e56c2c9f38eb177dc0
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 2b4165c106d05b6ef760b8e56c2c9f38eb177dc0
Author: Matt Crees <email address hidden>
Date: Wed Mar 27 16:15:54 2024 +0000

    RMQ: enable all stable feature flags at once

    Also enable these after an upgrade.

    Partial-Bug: #2058512
    Change-Id: Ib9bdae2e25c2b6cce30e4c8024015ab5875bc1ff
    (cherry picked from commit 223556e4e2efe35c77c4607408d47e67577a68f1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/922606
Committed: https://opendev.org/openstack/kolla-ansible/commit/b2a83c40ff0cdd8939f22d906b6d6cb46797584f
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit b2a83c40ff0cdd8939f22d906b6d6cb46797584f
Author: Matt Crees <email address hidden>
Date: Wed Mar 27 16:15:54 2024 +0000

    RMQ: enable all stable feature flags at once

    Also enable these after an upgrade.

    Partial-Bug: #2058512
    Change-Id: Ib9bdae2e25c2b6cce30e4c8024015ab5875bc1ff
    (cherry picked from commit 223556e4e2efe35c77c4607408d47e67577a68f1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/918975
Committed: https://opendev.org/openstack/kolla-ansible/commit/ef0785381b77919380bc5d9fde4b48fe167e0048
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ef0785381b77919380bc5d9fde4b48fe167e0048
Author: Matt Crees <email address hidden>
Date: Wed Mar 27 16:15:54 2024 +0000

    RMQ: enable all stable feature flags at once

    Also enable these after an upgrade.

    Partial-Bug: #2058512
    Change-Id: Ib9bdae2e25c2b6cce30e4c8024015ab5875bc1ff
    (cherry picked from commit b13964aecb33a68c0fc9939a8c720989bb7f958b)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.