Changing of pacemaker parameters breaks RabbitMQ test fail on parsing rabbitmqctl cluster_status output
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Nastya Urlapova | ||
8.0.x |
Fix Released
|
High
|
Nastya Urlapova |
Bug Description
Change pacemaker parameters doesn't break RabbitMQ.
Scenario:
1. Deploy environment with at least 3 controllers
2. Change max_rabbitmqctl
controllers,after that slaves rabbitmq will be restarted by
Pacemaker.
3. Wait for 1 minute.
4. Check RabbitMQ cluster is assembled until success in 10 min
5. Run OSTF
6. Repeat two more times steps 2-5
After attempt 5 next changes parameters crm_resource --resource p_rabbitmq-server --set-parameter max_rabbitmqctl
Cluster was not assembled in time timeout=600, interval=20, only master was running
2016-01-29 01:50:34,794 - DEBUG helpers.py:335 -- Executing command: 'rabbitmqctl cluster_status'
2016-01-29 01:50:36,000 - DEBUG __init__.py:59 -- Done: run_on_
2016-01-29 01:50:36,001 - INFO sftp.py:129 -- [chan 0] sftp session closed.
2016-01-29 01:50:36,064 - DEBUG test_failover_
{running_
After revert of failed env - looks like c cluster ok, on slaves node there are some errors like:
Error on AMQP connection <0.5460.0> (10.109.32.5:43207 -> 10.109.32.6:5673, vhost: '/', user: 'nova', state: running), channel 0:
{amqp_error,
"broker forced connection closure with reason 'shutdown'",none}
=ERROR REPORT==== 29-Jan-
Error on AMQP connection <0.3069.0> (10.109.32.6:48067 -> 10.109.32.6:5673, vhost: '/', user: 'nova', state: running), channel 0:
{amqp_error,
"broker forced connection closure with reason 'shutdown'",none}
[root@nailgun ~]# cat /etc/fuel/
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
api: "1.0"
build_number: "493"
build_id: "493"
fuel-nailgun_sha: "b900f9d9de4d2b
python-
fuel-agent_sha: "e869072139670b
fuel-
astute_sha: "b81577a5b7857c
fuel-library_sha: "f7a008e6801ba0
fuel-ostf_sha: "ab5fd151fc6c1a
fuel-mirror_sha: "351d568fa3b3e4
fuelmenu_sha: "fac143f4dfa757
shotgun_sha: "63645dea384a37
network-
fuel-upgrade_sha: "616a7490ec7199
fuelmain_sha: "6b993b3004e8d9
[root@nailgun ~]#
Changed in fuel: | |
status: | New → Confirmed |
summary: |
- Change pacemaker parameters break RabbitMQ on attemp 5 + Changing of pacemaker parameters breaks RabbitMQ on attempt 5 |
tags: | added: area-oslo |
tags: | added: move-to-mu |
summary: |
- Changing of pacemaker parameters breaks RabbitMQ on attempt 5 + Changing of pacemaker parameters breaks RabbitMQ test fail on parsing + rabbitmqctl cluster_status output |
Changed in fuel: | |
assignee: | Fuel QA Team (fuel-qa) → Artem Panchenko (apanchenko-8) |
status: | Confirmed → In Progress |
tags: | added: non-release |
Changed in fuel: | |
assignee: | Fuel QA Team (fuel-qa) → Nastya Urlapova (aurlapova) |
Changed in fuel: | |
status: | Confirmed → In Progress |
Changed in fuel: | |
status: | In Progress → Fix Committed |
tags: | removed: non-release |
As a side note - pacemaker logs contains a lot of messages from https:/ /bugs.launchpad .net/fuel/ +bug/1528686, but I don't think it's related to current bug.