RabbitMQ loses murano user when controller is offline

[Sergey Murashov]

ISO: {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Steps to reproduce:
1. Install OS+Murano
2. Run OSTF
OSTF would pass
3. Shutdown primary controller

Actual result:
Murano on remaining controllers would not able be to connect to infrastucture rabbit. Murano user wouldn't be in infrastucture rabbit

[Timur Nurlygayanov (tnurlygayanov)]

Looks like we can lost RabbitMQ users when we shutdown active node in HA cluster.

[Igor Yozhikov (iyozhikov)]

In case of one of rabbitmq cluster nodes went out(reboot, power off htc) rabbitmq user for murano disappears from cluster.

[Timur Nurlygayanov (tnurlygayanov)]

as a workaround we should add murano user in rabbitmq config or add murano user manually if we will hit this issue.

[Igor Yozhikov (iyozhikov)]

Workaround:
Use default rabbitmq credentials as used by other OS services, for example like nova or neutron.

[Sergey Murashov (smurashov)]

as workaround we can use default user for murano

[Igor Yozhikov (iyozhikov)]

Workaround follow-up:
1. Obtain of OS rabbitmq credentials
# grep -E "(^rabbit_user|^rabbit_pass)" /etc/nova/nova.conf
rabbit_userid=USERNAME
rabbit_password=SOMEPASS

2. Change /etc/murano/murano.conf DEFAULT section on all controllers in deployed environment with obtained values.
[DEFAULT]
...
rabbit_userid=USERNAME
rabbit_password=SOMEPASS
...

3. Restart murano-api and murano-engine services on all controllers in deployed environment
Ubuntu:
service murano-api restart
service murano-engine restart
CentOS:
service openstack-murano-api restart
service openstack-murano-engine restart

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/129676

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/129681

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/5.1
Review: https://review.openstack.org/129877

[OpenStack Infra (hudson-openstack)]

Change abandoned by Igor Yozhikov (<email address hidden>) on branch: stable/5.1
Review: https://review.openstack.org/129681
Reason: because of cherry-pick from master

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/129676
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6685028a3f540f0b27ace000fc287cc4bb7883f5
Submitter: Jenkins
Branch: master

commit 6685028a3f540f0b27ace000fc287cc4bb7883f5
Author: Igor Yozhikov <email address hidden>
Date: Mon Oct 20 21:10:56 2014 +0400

    Murano should use default rabbit credentials

    Change-Id: I45e9bf49ead2157ac763089e85eca1b43d8cd53d
    Related-Bug: #1372483

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/129877
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9c7c696bc3ac4c8aa36738048de73099bf32f603
Submitter: Jenkins
Branch: stable/5.1

commit 9c7c696bc3ac4c8aa36738048de73099bf32f603
Author: Igor Yozhikov <email address hidden>
Date: Mon Oct 20 21:10:56 2014 +0400

    Murano should use default rabbit credentials

    Change-Id: I45e9bf49ead2157ac763089e85eca1b43d8cd53d
    Related-Bug: #1372483

[Dmitry Borodaenko (angdraug)]

With the fixes merged as per comments above, what else remains to be done before this bug can be closed?

[Igor Yozhikov (iyozhikov)]

I made and fix another one bug - https://bugs.launchpad.net/fuel/+bug/1383765 while was fixing this one.
So, I believe that these bugs(1372483,1383765) could be closed after positive tests results.

[Stanislav Makar (smakar)]

checking on 6.0

[Stanislav Makar (smakar)]

{u'build_id': u'2014-12-09_22-41-06', u'ostf_sha': u'a9afb68710d809570460c29d6c3293219d3624d4', u'build_number': u'49', u'auth_required': True, u'nailgun_sha': u'22bd43b89a17843f9199f92d61fc86cb0f8772f1', u'production': u'docker', u'api': u'1.0', u'fuelmain_sha': u'3aab16667f47dd8384904e27f70f7a87ba15f4ee', u'astute_sha': u'16b252d93be6aaa73030b8100cf8c5ca6a970a91', u'feature_groups': [u'mirantis'], u'release': u'6.0', u'release_versions': {u'2014.2-6.0': {u'VERSION': {u'build_id': u'2014-12-09_22-41-06', u'ostf_sha': u'a9afb68710d809570460c29d6c3293219d3624d4', u'build_number': u'49', u'api': u'1.0', u'nailgun_sha': u'22bd43b89a17843f9199f92d61fc86cb0f8772f1', u'production': u'docker', u'fuelmain_sha': u'3aab16667f47dd8384904e27f70f7a87ba15f4ee', u'astute_sha': u'16b252d93be6aaa73030b8100cf8c5ca6a970a91', u'feature_groups': [u'mirantis'], u'release': u'6.0', u'fuellib_sha': u'2c99931072d951301d395ebd5bf45c8d401301bb'}}}, u'fuellib_sha': u'2c99931072d951301d395ebd5bf45c8d401301bb'}