RabbitMQ do not keep non-default users/vhosts/etc after failover

Bug #1383258 reported by Vladimir Kuklin
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Alex Schultz
6.1.x
Won't Fix
High
MOS Maintenance
7.0.x
Fix Released
High
Alex Schultz

Bug Description

We need to dump rabbit users, such as murano to separate file and recreate them using OCF script in case of failover.

Stanislav Makar (smakar)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislav Makar (smakar)
Stanislav Makar (smakar)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/132967

Revision history for this message
Matthew Mosesohn (raytrac3r) wrote : Re: RabbitMQ does not keep non-default users after failover

Lowering priority because there is a workaround and it is not an easy fix with syncing upstream manifests.

Changed in fuel:
importance: High → Medium
milestone: 6.0 → 6.1
Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote :

Stanislav, any updates? This bug hang there about 3 months now, are you really working on it?

Stanislav Makar (smakar)
Changed in fuel:
status: In Progress → Confirmed
Changed in fuel:
importance: Medium → High
Changed in fuel:
status: Confirmed → In Progress
tags: added: customer-found
tags: added: docs release-notes
Stanislav Makar (smakar)
Changed in fuel:
status: In Progress → Won't Fix
Changed in fuel:
status: Won't Fix → Confirmed
status: Confirmed → Won't Fix
Revision history for this message
Stanislav Makar (smakar) wrote :

postponed to 7.0
there is no profit in 6.1

summary: - RabbitMQ does not keep non-default users after failover
+ RabbitMQ do not keep non-default users/vhosts/etc after failover
tags: added: release-notes-done
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (stable/6.1)

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/194961

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (stable/6.1)
Download full text (45.4 KiB)

Reviewed: https://review.openstack.org/194961
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=0e26e7d7cc153d179ec34985645dd23cdd239ddb
Submitter: Jenkins
Branch: stable/6.1

commit 5cc5f0c643aebecaf3bf4580535a3ea7c3334a6c
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:43:35 2015 -0700

    Removed streamlined patching backend pieces

    Change-Id: I955e76ccdbd12a9145f4e9b689f80bdf9fcaf929

commit 563c4b5c78ebfcb1f4f91047c2919f6270f9a1d4
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:30:30 2015 -0700

    Removed outdated patching guide

    Change-Id: I76180c277789ade9c5ebedd19fe2092847c0b7d9

commit 8d120c14bec1ab41d448683ad146a3053a57c4ee
Author: Irina Povolotskaya <email address hidden>
Date: Tue Jun 23 19:59:11 2015 +0300

    Add dual hypervisor ref arch into 6.1 docs

    Change-Id: I900c24c9de878eafadbfc995aa879b7f55737fac

commit feebd1592d3305b64bbdfd0bc5fe108190aef120
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:38:17 2015 +0300

    [OPs guide] Running Ceilometer section edits

    1. conf file extract is updated
    2. note is updated

    Closes-bug: 1467817
    Change-Id: I0217e164108e0ba6c1397045a5e57d13ff429223

commit 44a93f9dead7511a3461ec35248dbb689c81eafd
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:04:40 2015 +0300

    [RN6_1] Final changes

    1. capitalization
    2. 2014.2 to 2014.2.2
    3. general improvements

    Change-Id: I45057e90c90550559f66bc67ccdf97a559fd9000

commit bb41389cae58084285688853281516b659686422
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 16:45:35 2015 +0300

    Update patching decription

    Update patching description with
    the standard Linux commands.

    Change-Id: Ia1a8346639c468fdfce15a11d2430bf3a4731244

commit bf3018fae3f2e564413d33aba6cdebf8868f0b4e
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 15:55:49 2015 +0300

    [RN6_1] Clean up

    1. Rearranges sections
    2. Improves RST
    3. Changes titles order

    Change-Id: I6110bf515667d3d6ba08ad35ff5d593dbc96641e

commit 1c7e4457808e8f2d6c56fdf31252170972e444b9
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 15:26:28 2015 +0300

    Replaces VBOX screenshots

    This patch:
    - replaces VBOX screenshots
    - changes the link for Download Mirantis VirtualBox scripts
     to https://docs.mirantis.com/openstack/fuel/fuel-master/#downloads

    Change-Id: I58dede960c5c3355d39b07ff44b757403f6af02c
    Closes-Bug: #1467872

commit 0a568bf53fc0e25d1d692d5d74b4a7b4d983bbcc
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 14:01:55 2015 +0300

    6.1 --separate repos

    change wording and add links to the
    separate repos feature.

    Change-Id: Ib5d0778a0d8f1534f79ed2f553574cb69a3150b0

commit 95a188b21cbdd064d92696b7920e6a0105fe0c56
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 12:07:28 2015 +0300

    Corrects the output 'pcs status'

    Changes the example outputs to appropriate ones.

    Change-Id: Ib6d83...

Changed in fuel:
status: Won't Fix → In Progress
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Please extend this case to custom virtual hosts as well

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/132967
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8ccdfc62152de8212ba6a0aeed45c1e3677c70a4
Submitter: Jenkins
Branch: master

commit 8ccdfc62152de8212ba6a0aeed45c1e3677c70a4
Author: Stanislav Makar <email address hidden>
Date: Thu Apr 16 14:32:51 2015 +0000

    Implement the dumping of rabbitMQ definitions

    This changes leverages the rabbitmq management plugin to dump
    exchanges, queues, bindings, users, virtual hosts, permissions and
    parameters from the running system. Specifically this change adds the
    following:

    * The dumping rabbitMQ definitions (users/vhosts/exchanges/etc) during
      the end of the deployment
    * The possibility to restore definitions to the rabbitmq-server ocf
      script during rabbitMQ startup.
    * Enabled rabbitmq admin plugin, but restricts it to localhost traffic.
      This reverts Ic01c26200f6019a8112b1c5fb04a282e64b3b3e6 but adds
      firewall rules to mitigate the issue.

    DocImpact: The dump_rabbit_definitions task can be used to backup the
    rabbitmq definitions and if custom definitions (users/vhosts/etc) are
    created it must be run or the changes may be lost during the rabbitmq
    failover via pacemaker.

    Change-Id: I715f7c2ae527f7e105b9f6b7d82c443e8accf178
    Closes-bug: #1383258
    Related-bug: #1450443
    Co-Authored-By: Alex Schultz <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Reopening as for some cases, the function fails with
err: ERROR: /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs: line 379: /usr/bin/curl: Argument list too long
err: ERROR: RMQ definitions have not imported.

The scale lab team has more details. This issue may be seen with large numbers of queues, like 5000 or more

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/212555

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/212555
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=11c4e4c3985949f776367dd5045102c6398aef89
Submitter: Jenkins
Branch: master

commit 11c4e4c3985949f776367dd5045102c6398aef89
Author: Alex Schultz <email address hidden>
Date: Thu Aug 13 09:31:51 2015 -0500

    Fix rabbitmq data restore for large datasets

    Previously we were sending the json backup data on the command line
    which fails when the dataset is large. This change updates the command
    line options for curl to pass the filename directly and let it handle
    the reading of the data.

    Change-Id: I37f298279beca06df41fb08e1745602976c6a776
    Closes-Bug: 1383258

Changed in fuel:
status: In Progress → Fix Committed
oleksii shyman (oshyman)
tags: added: on-validation
oleksii shyman (oshyman)
tags: added: scale
removed: on-validation
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/223470

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (master)

Reviewed: https://review.openstack.org/223470
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=38c61bb917bb127085f785a386a40183df2679f5
Submitter: Jenkins
Branch: master

commit 38c61bb917bb127085f785a386a40183df2679f5
Author: evkonstantinov <email address hidden>
Date: Tue Sep 15 11:00:45 2015 +0300

    Add rabbit definitions dump resolved issue to relnotes

    Change-Id: Ic9ef9840b884a44339a3ae3a9def38cb2eb1f5f1
    Related-Bug:#1383258

tags: added: on-verification
Revision history for this message
Vasiliy Pleshakov (vpleshakov) wrote :

Verified with fuel-7.0-256-2015-08-30_09-35-52.iso

[root@fuel ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "256"
  build_id: "2015-08-30_09-35-52"
  nailgun_sha: "3189ccfb8c1dac888e351f535b03bdbc9d392406"
  python-fuelclient_sha: "9643fa07f1290071511066804f962f62fe27b512"
  fuel-agent_sha: "1e8f38bbb864ed99aa8fe862b6367e82afec3263"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "53c86cba593ddbac776ce5a3360240274c20738c"
  fuel-library_sha: "bc04a7092d92400c79e6ea6ede25e7b67c6a6355"
  fuel-ostf_sha: "644db51186dc23c9b27e9b5486c120c8363dc87c"
  fuelmain_sha: "0e54d68392b359bc122e5bbba9249c729eeaf579"

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.