| 2015-07-07 11:24:03 |
Tatyanka |
bug |
|
|
added bug |
| 2015-07-07 11:24:03 |
Tatyanka |
attachment added |
|
fuel-snapshot-2015-07-07_10-40-09.tar.xz https://bugs.launchpad.net/bugs/1472230/+attachment/4425724/+files/fuel-snapshot-2015-07-07_10-40-09.tar.xz |
|
| 2015-07-07 11:25:53 |
Tatyanka |
summary |
Pecemaker shows healty status fro rabbitmq meanwhile onde node of rabbitmq failed |
Pecemaker shows healthy status fro rabbitmq meanwhile onั node of rabbitmq failed |
|
| 2015-07-07 11:28:23 |
Tatyanka |
description |
http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.cic_maintenance_mode/18/testReport/junit/(root)/auto_cic_maintenance_mode/auto_cic_maintenance_mode/
๐๐ญ๐๐ฉ๐ฌ:
1. Create cluster
2. Add 3 node with controller and mongo roles
3. Add 2 node with compute and cinder roles
4. Deploy the cluster
5. Run ostf
6. Check that umm feature is enabled (umm status )
7. Call unexpected reboot: reboot --force >/dev/null &
8. Check that node is rebooted, back online and enter into auto mode(umm status):
9. Disable umm mode: umm off and wait umm stops
10. Check while node back to the online status in psc
11. Run ostf ha
๐๐๐ญ๐ฎ๐๐ฅ ๐ซ๐๐ฌ๐ฎ๐ฅ๐ญ:
OSTF rabbit tests are failed:
Failed 2 OSTF tests; should fail 0 tests. Names of failed tests: [{u'RabbitMQ availability (failure)': u'Number of controllers is not equal to number of cluster nodes.'}, {u'RabbitMQ replication (failure)': u'Failed to connect to 5673 port on host 10.109.20.4 Please refer to OpenStack logs for more details.'}]
rabbit failed node is node-2:
๐๐ฅ๐ฎ๐ฌ๐ญ๐๐ซ ๐ฌ๐ญ๐๐ญ๐ฎ๐ฌ ๐จ๐ ๐ง๐จ๐๐ '๐ซ๐๐๐๐ข๐ญ@๐ง๐จ๐๐-๐' ...
Error: unable to connect to node 'rabbit@node-2': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit@node-2']
rabbit@node-2:
* connected to epmd (port 4369) on node-2
* epmd reports: node 'rabbit' not running at all
no other nodes on node-2
* suggestion: start the node
current node details:
- node name: 'rabbitmqctl11900@node-2'
- home dir: /var/lib/rabbitmq
- cookie hash: soeIWU2jk2YNseTyDSlsEA==
๐๐ง๐ ๐ฌ๐๐๐ฆ๐ฌ ๐ซ๐๐๐๐ข๐ญ ๐ข๐ฌ ๐ง๐จ๐ญ ๐ซ๐ฎ๐ง๐ง๐ข๐ง๐ ๐จ๐ง ๐ข๐ญ:
root@node-2:~# ps uuax| grep erla
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 14648 0.0 0.0 10460 936 pts/0 S+ 10:23 0:00 grep --color=auto erla
root@node-2:~# ps uuax| grep beam
root 14729 0.0 0.0 10460 936 pts/0 S+ 10:24 0:00 grep --color=auto beam
root@node-2:~# ps uuax| grep rabb
rabbitmq 3438 0.0 0.4 90432 11844 ? Ss 09:00 0:00 /usr/bin/python /usr/bin/rabbit-fence.py
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
๐๐ง๐ ๐จ๐๐ ๐ญ๐จ๐จ:
root@node-2:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/rabbitmq-server status ; echo $?
7
๐๐ฎ๐ญ ๐ฉ๐๐๐๐ฆ๐๐ค๐๐ซ ๐ฌ๐ก๐จ๐ฐ ๐ข๐ญ ๐๐ฌ ๐ก๐๐๐ฅ๐ญ๐ก๐ฒ ๐๐ง๐ ๐จ๐ง๐ฅ๐ข๐ง๐ ๐๐ง๐ ๐๐ฏ๐๐ง ๐๐จ ๐ง๐จ ๐ญ๐ซ๐ฒ ๐ญ๐จ ๐ซ๐-๐ฎ๐ฉ:
Online: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Full list of resources:
Clone Set: clone_p_vrouter [p_vrouter]
Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
vip__management (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
vip__public_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
vip__management_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
vip__public (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
Master/Slave Set: master_p_conntrackd [p_conntrackd]
Masters: [ node-1.test.domain.local ]
Slaves: [ node-2.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_haproxy [p_haproxy]
Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_dns [p_dns]
Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_mysql [p_mysql]
Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
p_ceilometer-agent-central (ocf::fuel:ceilometer-agent-central): Started node-1.test.domain.local
p_ceilometer-alarm-evaluator (ocf::fuel:ceilometer-alarm-evaluator): Started node-2.test.domain.local
๐ด๐๐๐๐๐/๐บ๐๐๐๐ ๐บ๐๐: ๐๐๐๐๐๐_๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐ [๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐]
๐ด๐๐๐๐๐๐: [ ๐๐๐
๐-1.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
๐บ๐๐๐๐๐: [ ๐๐๐
๐-2.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ๐๐๐
๐-5.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
Clone Set: clone_p_heat-engine [p_heat-engine]
Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_ntp [p_ntp]
Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Clone Set: clone_ping_vip__public [ping_vip__public]
Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
PCSD Status:
10.109.22.4: Offline
10.109.22.5: Offline
10.109.22.8: Offline |
http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.cic_maintenance_mode/18/testReport/junit/(root)/auto_cic_maintenance_mode/auto_cic_maintenance_mode/
๐๐ญ๐๐ฉ๐ฌ:
1. Create cluster
2. Add 3 node with controller and mongo roles
3. Add 2 node with compute and cinder roles
4. Deploy the cluster
5. Run ostf
6. Check that umm feature is enabled (umm status )
7. Call unexpected reboot: reboot --force >/dev/null &
8. Check that node is rebooted, back online and enter into auto mode(umm status):
9. Disable umm mode: umm off and wait umm stops
10. Check while node back to the online status in psc
11. Run ostf ha
๐๐๐ญ๐ฎ๐๐ฅ ๐ซ๐๐ฌ๐ฎ๐ฅ๐ญ:
OSTF rabbit tests are failed:
Failed 2 OSTF tests; should fail 0 tests. Names of failed tests: [{u'RabbitMQ availability (failure)': u'Number of controllers is not equal to number of cluster nodes.'}, {u'RabbitMQ replication (failure)': u'Failed to connect to 5673 port on host 10.109.20.4 Please refer to OpenStack logs for more details.'}]
rabbit failed node is node-2:
๐๐ฅ๐ฎ๐ฌ๐ญ๐๐ซ ๐ฌ๐ญ๐๐ญ๐ฎ๐ฌ ๐จ๐ ๐ง๐จ๐๐ '๐ซ๐๐๐๐ข๐ญ@๐ง๐จ๐๐-๐' ...
Error: unable to connect to node 'rabbit@node-2': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit@node-2']
rabbit@node-2:
ย ย * connected to epmd (port 4369) on node-2
ย ย * epmd reports: node 'rabbit' not running at all
ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย no other nodes on node-2
ย ย * suggestion: start the node
current node details:
- node name: 'rabbitmqctl11900@node-2'
- home dir: /var/lib/rabbitmq
- cookie hash: soeIWU2jk2YNseTyDSlsEA==
๐๐ง๐ ๐ฌ๐๐๐ฆ๐ฌ ๐ซ๐๐๐๐ข๐ญ ๐ข๐ฌ ๐ง๐จ๐ญ ๐ซ๐ฎ๐ง๐ง๐ข๐ง๐ ๐จ๐ง ๐ข๐ญ:
root@node-2:~# ps uuax| grep erla
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 14648 0.0 0.0 10460 936 pts/0 S+ 10:23 0:00 grep --color=auto erla
root@node-2:~# ps uuax| grep beam
root 14729 0.0 0.0 10460 936 pts/0 S+ 10:24 0:00 grep --color=auto beam
root@node-2:~# ps uuax| grep rabb
rabbitmq 3438 0.0 0.4 90432 11844 ? Ss 09:00 0:00 /usr/bin/python /usr/bin/rabbit-fence.py
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
๐๐ง๐ ๐จ๐๐ ๐ญ๐จ๐จ:
root@node-2:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/rabbitmq-server status ; echo $?
7
๐๐ฎ๐ญ ๐ฉ๐๐๐๐ฆ๐๐ค๐๐ซ ๐ฌ๐ก๐จ๐ฐ ๐ข๐ญ ๐๐ฌ ๐ก๐๐๐ฅ๐ญ๐ก๐ฒ ๐๐ง๐ ๐จ๐ง๐ฅ๐ข๐ง๐ ๐๐ง๐ ๐๐ฏ๐๐ง ๐๐จ ๐ง๐จ ๐ญ๐ซ๐ฒ ๐ญ๐จ ๐ซ๐-๐ฎ๐ฉ:
Online: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Full list of resources:
ย Clone Set: clone_p_vrouter [p_vrouter]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย vip__management (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__public_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__management_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__public (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย Master/Slave Set: master_p_conntrackd [p_conntrackd]
ย ย ย ย ย Masters: [ node-1.test.domain.local ]
ย ย ย ย ย Slaves: [ node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_haproxy [p_haproxy]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_dns [p_dns]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_mysql [p_mysql]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย p_ceilometer-agent-central (ocf::fuel:ceilometer-agent-central): Started node-1.test.domain.local
ย p_ceilometer-alarm-evaluator (ocf::fuel:ceilometer-alarm-evaluator): Started node-2.test.domain.local
ย ๐ด๐๐๐๐๐/๐บ๐๐๐๐ ๐บ๐๐: ๐๐๐๐๐๐_๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐ [๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐]
ย ย ย ย ๐ด๐๐๐๐๐๐: [ ๐๐๐
๐-1.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
ย ย ย ๐บ๐๐๐๐๐: [ ๐๐๐
๐-2.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ๐๐๐
๐-5.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
Clone Set: clone_p_heat-engine [p_heat-engine]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_ntp [p_ntp]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_ping_vip__public [ping_vip__public]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
PCSD Status:
ย ย 10.109.22.4: Offline
ย ย 10.109.22.5: Offline
ย ย 10.109.22.8: Offline
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "7.0"
openstack_version: "2014.2.2-7.0"
api: "1.0"
build_number: "26"
build_id: "2015-07-06_18-08-24"
nailgun_sha: "d040c5cebc9cdd24ef20cb7ecf0a337039baddec"
python-fuelclient_sha: "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45"
astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
fuel-library_sha: "251c54e8de2f41aacd260751e7a891e9fbffc45d"
fuel-ostf_sha: "a752c857deafd2629baf646b1b3188f02ff38084"
fuelmain_sha: "4f2dff3bdc327858fa45bcc2853cfbceae68a40c" |
|
| 2015-07-07 11:39:49 |
OpenStack Infra |
fuel: status |
New |
In Progress |
|
| 2015-07-07 11:39:49 |
OpenStack Infra |
fuel: assignee |
Fuel Library Team (fuel-library) |
Bogdan Dobrelya (bogdando) |
|
| 2015-07-07 16:10:59 |
Tatyanka |
description |
http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.cic_maintenance_mode/18/testReport/junit/(root)/auto_cic_maintenance_mode/auto_cic_maintenance_mode/
๐๐ญ๐๐ฉ๐ฌ:
1. Create cluster
2. Add 3 node with controller and mongo roles
3. Add 2 node with compute and cinder roles
4. Deploy the cluster
5. Run ostf
6. Check that umm feature is enabled (umm status )
7. Call unexpected reboot: reboot --force >/dev/null &
8. Check that node is rebooted, back online and enter into auto mode(umm status):
9. Disable umm mode: umm off and wait umm stops
10. Check while node back to the online status in psc
11. Run ostf ha
๐๐๐ญ๐ฎ๐๐ฅ ๐ซ๐๐ฌ๐ฎ๐ฅ๐ญ:
OSTF rabbit tests are failed:
Failed 2 OSTF tests; should fail 0 tests. Names of failed tests: [{u'RabbitMQ availability (failure)': u'Number of controllers is not equal to number of cluster nodes.'}, {u'RabbitMQ replication (failure)': u'Failed to connect to 5673 port on host 10.109.20.4 Please refer to OpenStack logs for more details.'}]
rabbit failed node is node-2:
๐๐ฅ๐ฎ๐ฌ๐ญ๐๐ซ ๐ฌ๐ญ๐๐ญ๐ฎ๐ฌ ๐จ๐ ๐ง๐จ๐๐ '๐ซ๐๐๐๐ข๐ญ@๐ง๐จ๐๐-๐' ...
Error: unable to connect to node 'rabbit@node-2': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit@node-2']
rabbit@node-2:
ย ย * connected to epmd (port 4369) on node-2
ย ย * epmd reports: node 'rabbit' not running at all
ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย no other nodes on node-2
ย ย * suggestion: start the node
current node details:
- node name: 'rabbitmqctl11900@node-2'
- home dir: /var/lib/rabbitmq
- cookie hash: soeIWU2jk2YNseTyDSlsEA==
๐๐ง๐ ๐ฌ๐๐๐ฆ๐ฌ ๐ซ๐๐๐๐ข๐ญ ๐ข๐ฌ ๐ง๐จ๐ญ ๐ซ๐ฎ๐ง๐ง๐ข๐ง๐ ๐จ๐ง ๐ข๐ญ:
root@node-2:~# ps uuax| grep erla
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 14648 0.0 0.0 10460 936 pts/0 S+ 10:23 0:00 grep --color=auto erla
root@node-2:~# ps uuax| grep beam
root 14729 0.0 0.0 10460 936 pts/0 S+ 10:24 0:00 grep --color=auto beam
root@node-2:~# ps uuax| grep rabb
rabbitmq 3438 0.0 0.4 90432 11844 ? Ss 09:00 0:00 /usr/bin/python /usr/bin/rabbit-fence.py
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
๐๐ง๐ ๐จ๐๐ ๐ญ๐จ๐จ:
root@node-2:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/rabbitmq-server status ; echo $?
7
๐๐ฎ๐ญ ๐ฉ๐๐๐๐ฆ๐๐ค๐๐ซ ๐ฌ๐ก๐จ๐ฐ ๐ข๐ญ ๐๐ฌ ๐ก๐๐๐ฅ๐ญ๐ก๐ฒ ๐๐ง๐ ๐จ๐ง๐ฅ๐ข๐ง๐ ๐๐ง๐ ๐๐ฏ๐๐ง ๐๐จ ๐ง๐จ ๐ญ๐ซ๐ฒ ๐ญ๐จ ๐ซ๐-๐ฎ๐ฉ:
Online: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Full list of resources:
ย Clone Set: clone_p_vrouter [p_vrouter]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย vip__management (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__public_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__management_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__public (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย Master/Slave Set: master_p_conntrackd [p_conntrackd]
ย ย ย ย ย Masters: [ node-1.test.domain.local ]
ย ย ย ย ย Slaves: [ node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_haproxy [p_haproxy]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_dns [p_dns]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_mysql [p_mysql]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย p_ceilometer-agent-central (ocf::fuel:ceilometer-agent-central): Started node-1.test.domain.local
ย p_ceilometer-alarm-evaluator (ocf::fuel:ceilometer-alarm-evaluator): Started node-2.test.domain.local
ย ๐ด๐๐๐๐๐/๐บ๐๐๐๐ ๐บ๐๐: ๐๐๐๐๐๐_๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐ [๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐]
ย ย ย ย ๐ด๐๐๐๐๐๐: [ ๐๐๐
๐-1.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
ย ย ย ๐บ๐๐๐๐๐: [ ๐๐๐
๐-2.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ๐๐๐
๐-5.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
Clone Set: clone_p_heat-engine [p_heat-engine]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_ntp [p_ntp]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_ping_vip__public [ping_vip__public]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
PCSD Status:
ย ย 10.109.22.4: Offline
ย ย 10.109.22.5: Offline
ย ย 10.109.22.8: Offline
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "7.0"
openstack_version: "2014.2.2-7.0"
api: "1.0"
build_number: "26"
build_id: "2015-07-06_18-08-24"
nailgun_sha: "d040c5cebc9cdd24ef20cb7ecf0a337039baddec"
python-fuelclient_sha: "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45"
astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
fuel-library_sha: "251c54e8de2f41aacd260751e7a891e9fbffc45d"
fuel-ostf_sha: "a752c857deafd2629baf646b1b3188f02ff38084"
fuelmain_sha: "4f2dff3bdc327858fa45bcc2853cfbceae68a40c" |
http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.cic_maintenance_mode/18/testReport/junit/(root)/auto_cic_maintenance_mode/auto_cic_maintenance_mode/
๐๐ญ๐๐ฉ๐ฌ:
1. Create cluster
2. Add 3 node with controller and mongo roles
3. Add 2 node with compute and cinder roles
4. Deploy the cluster
5. Run ostf
6. Check that umm feature is enabled (umm status )
7. Call unexpected reboot: reboot --force >/dev/null &
8. Check that node is rebooted, back online and enter into auto mode(umm status):
9. Disable umm mode: umm off and wait umm stops
10. Check while node back to the online status in psc
11. Run ostf ha
๐๐๐ญ๐ฎ๐๐ฅ ๐ซ๐๐ฌ๐ฎ๐ฅ๐ญ:
OSTF rabbit tests are failed:
Failed 2 OSTF tests; should fail 0 tests. Names of failed tests: [{u'RabbitMQ availability (failure)': u'Number of controllers is not equal to number of cluster nodes.'}, {u'RabbitMQ replication (failure)': u'Failed to connect to 5673 port on host 10.109.20.4 Please refer to OpenStack logs for more details.'}]
rabbit failed node is node-2:
๐๐ฅ๐ฎ๐ฌ๐ญ๐๐ซ ๐ฌ๐ญ๐๐ญ๐ฎ๐ฌ ๐จ๐ ๐ง๐จ๐๐ '๐ซ๐๐๐๐ข๐ญ@๐ง๐จ๐๐-๐' ...
Error: unable to connect to node 'rabbit@node-2': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit@node-2']
rabbit@node-2:
ย ย * connected to epmd (port 4369) on node-2
ย ย * epmd reports: node 'rabbit' not running at all
ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย ย no other nodes on node-2
ย ย * suggestion: start the node
current node details:
- node name: 'rabbitmqctl11900@node-2'
- home dir: /var/lib/rabbitmq
- cookie hash: soeIWU2jk2YNseTyDSlsEA==
๐๐ง๐ ๐ฌ๐๐๐ฆ๐ฌ ๐ซ๐๐๐๐ข๐ญ ๐ข๐ฌ ๐ง๐จ๐ญ ๐ซ๐ฎ๐ง๐ง๐ข๐ง๐ ๐จ๐ง ๐ข๐ญ:
root@node-2:~# ps uuax| grep erla
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 14648 0.0 0.0 10460 936 pts/0 S+ 10:23 0:00 grep --color=auto erla
root@node-2:~# ps uuax| grep beam
root 14729 0.0 0.0 10460 936 pts/0 S+ 10:24 0:00 grep --color=auto beam
root@node-2:~# ps uuax| grep rabb
rabbitmq 3438 0.0 0.4 90432 11844 ? Ss 09:00 0:00 /usr/bin/python /usr/bin/rabbit-fence.py
rabbitmq 8081 0.0 0.0 8132 1088 ? S 09:01 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
๐๐ง๐ ๐จ๐๐ ๐ญ๐จ๐จ:
root@node-2:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/rabbitmq-server status ; echo $?
7
๐๐ฎ๐ญ ๐ฉ๐๐๐๐ฆ๐๐ค๐๐ซ ๐ฌ๐ก๐จ๐ฐ ๐ข๐ญ ๐๐ฌ ๐ก๐๐๐ฅ๐ญ๐ก๐ฒ ๐๐ง๐ ๐จ๐ง๐ฅ๐ข๐ง๐ ๐๐ง๐ ๐๐ฏ๐๐ง ๐๐จ ๐ง๐จ ๐ญ๐ซ๐ฒ ๐ญ๐จ ๐ซ๐-๐ฎ๐ฉ:
Online: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
Full list of resources:
ย Clone Set: clone_p_vrouter [p_vrouter]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย vip__management (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__public_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__management_vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย vip__public (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
ย Master/Slave Set: master_p_conntrackd [p_conntrackd]
ย ย ย ย ย Masters: [ node-1.test.domain.local ]
ย ย ย ย ย Slaves: [ node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_haproxy [p_haproxy]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_dns [p_dns]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_mysql [p_mysql]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย p_ceilometer-agent-central (ocf::fuel:ceilometer-agent-central): Started node-1.test.domain.local
ย p_ceilometer-alarm-evaluator (ocf::fuel:ceilometer-alarm-evaluator): Started node-2.test.domain.local
ย ๐ด๐๐๐๐๐/๐บ๐๐๐๐ ๐บ๐๐: ๐๐๐๐๐๐_๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐ [๐_๐๐๐๐๐๐๐๐-๐๐๐๐๐๐]
ย ย ย ย ๐ด๐๐๐๐๐๐: [ ๐๐๐
๐-1.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
ย ย ย ๐บ๐๐๐๐๐: [ ๐๐๐
๐-2.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ๐๐๐
๐-5.๐๐๐๐.๐
๐๐๐๐๐.๐๐๐๐๐ ]
Clone Set: clone_p_heat-engine [p_heat-engine]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_p_ntp [p_ntp]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
ย Clone Set: clone_ping_vip__public [ping_vip__public]
ย ย ย ย ย Started: [ node-1.test.domain.local node-2.test.domain.local node-5.test.domain.local ]
PCSD Status:
ย ย 10.109.22.4: Offline
ย ย 10.109.22.5: Offline
ย ย 10.109.22.8: Offline
VERSION:
ย ย feature_groups:
ย ย ย ย - mirantis
ย ย production: "docker"
ย ย release: "7.0"
ย ย openstack_version: "2014.2.2-7.0"
ย ย api: "1.0"
ย ย build_number: "26"
ย ย build_id: "2015-07-06_18-08-24"
ย ย nailgun_sha: "d040c5cebc9cdd24ef20cb7ecf0a337039baddec"
ย ย python-fuelclient_sha: "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45"
ย ย astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
ย ย fuel-library_sha: "251c54e8de2f41aacd260751e7a891e9fbffc45d"
ย ย fuel-ostf_sha: "a752c857deafd2629baf646b1b3188f02ff38084"
ย ย fuelmain_sha: "4f2dff3bdc327858fa45bcc2853cfbceae68a40c" |
|
| 2015-07-08 12:53:14 |
Bogdan Dobrelya |
bug |
|
|
added subscriber Davanum Srinivas (DIMS) |
| 2015-07-08 12:54:02 |
Bogdan Dobrelya |
summary |
Pecemaker shows healthy status fro rabbitmq meanwhile onั node of rabbitmq failed |
Pacemaker shows healthy status for rabbitmq node meanwhile the node is actually down |
|
| 2015-07-08 12:54:17 |
Bogdan Dobrelya |
tags |
pacemaker |
pacemaker rabbitmq |
|
| 2015-07-08 22:01:49 |
Mike Scherbakov |
bug |
|
|
added subscriber Mike Scherbakov |
| 2015-07-09 07:26:42 |
Bogdan Dobrelya |
fuel: status |
In Progress |
Fix Committed |
|
| 2015-08-07 10:00:56 |
Tatyanka |
fuel: status |
Fix Committed |
Fix Released |
|
| 2015-08-27 12:38:05 |
Bogdan Dobrelya |
fuel: status |
Fix Released |
Confirmed |
|
| 2015-08-27 12:44:26 |
Bogdan Dobrelya |
summary |
Pacemaker shows healthy status for rabbitmq node meanwhile the node is actually down |
Pacemaker shows healthy status for rabbitmq node meanwhile the node is actually down/split brain |
|
| 2015-08-27 12:57:32 |
Bogdan Dobrelya |
attachment added |
|
node-6_pacemaker.tgz https://bugs.launchpad.net/fuel/+bug/1472230/+attachment/4453328/+files/node-6_pacemaker.tgz |
|
| 2015-08-27 15:03:24 |
Bogdan Dobrelya |
attachment added |
|
events_around.tgz https://bugs.launchpad.net/fuel/+bug/1472230/+attachment/4453402/+files/events_around.tgz |
|
| 2015-08-27 15:05:18 |
Bogdan Dobrelya |
fuel: status |
Confirmed |
Fix Released |
|
| 2015-09-15 14:35:31 |
Bogdan Dobrelya |
fuel: status |
Fix Released |
Fix Committed |
|
| 2015-09-17 11:13:31 |
Artem Hrechanychenko |
tags |
pacemaker rabbitmq |
on-verification pacemaker rabbitmq |
|
| 2015-09-17 20:16:20 |
Artem Hrechanychenko |
fuel: status |
Fix Committed |
Fix Released |
|
| 2015-09-21 12:41:56 |
Artem Panchenko |
nominated for series |
|
fuel/8.0.x |
|
| 2015-09-21 12:41:56 |
Artem Panchenko |
bug task added |
|
fuel/8.0.x |
|
| 2015-09-21 12:41:56 |
Artem Panchenko |
nominated for series |
|
fuel/7.0.x |
|
| 2015-09-21 12:41:56 |
Artem Panchenko |
bug task added |
|
fuel/7.0.x |
|
| 2015-09-21 12:42:05 |
Artem Panchenko |
fuel/7.0.x: importance |
Undecided |
High |
|
| 2015-09-21 12:42:17 |
Artem Panchenko |
fuel/8.0.x: milestone |
7.0 |
8.0 |
|
| 2015-09-21 12:42:53 |
Artem Panchenko |
fuel/7.0.x: milestone |
|
7.0 |
|
| 2015-09-21 12:54:51 |
Alexander Gordeev |
fuel/7.0.x: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2015-09-21 12:55:45 |
Artem Panchenko |
fuel/7.0.x: assignee |
Fuel Library Team (fuel-library) |
Vladimir Kuklin (vkuklin) |
|
| 2015-09-21 12:55:48 |
Artem Panchenko |
fuel/7.0.x: status |
New |
In Progress |
|
| 2015-09-21 12:56:01 |
Artem Panchenko |
fuel/8.0.x: status |
Fix Released |
Fix Committed |
|
| 2015-09-21 13:12:15 |
Nastya Urlapova |
fuel/7.0.x: importance |
High |
Critical |
|
| 2015-09-21 15:58:19 |
Andrey Maximov |
fuel/7.0.x: assignee |
Vladimir Kuklin (vkuklin) |
Sergii Golovatiuk (sgolovatiuk) |
|
| 2015-09-22 09:26:43 |
Nastya Urlapova |
fuel/7.0.x: status |
In Progress |
Fix Committed |
|
| 2015-09-24 13:48:27 |
Dennis Dmitriev |
tags |
on-verification pacemaker rabbitmq |
pacemaker rabbitmq |
|
| 2015-09-24 13:48:32 |
Dennis Dmitriev |
tags |
pacemaker rabbitmq |
on-verification pacemaker rabbitmq |
|
| 2015-09-25 13:22:49 |
Dennis Dmitriev |
tags |
on-verification pacemaker rabbitmq |
pacemaker rabbitmq |
|
| 2015-09-25 13:22:54 |
Dennis Dmitriev |
fuel/7.0.x: status |
Fix Committed |
Fix Released |
|
| 2015-09-25 13:22:58 |
Dennis Dmitriev |
fuel/8.0.x: status |
Fix Committed |
Fix Released |
|
| 2015-10-22 03:15:21 |
Dmitry Pyzhov |
tags |
pacemaker rabbitmq |
area-library pacemaker rabbitmq |
|
| 2015-11-05 11:45:02 |
Sergey Shevorakov |
tags |
area-library pacemaker rabbitmq |
area-library pacemaker rabbitmq rca-done |
|
| 2015-12-31 09:17:26 |
Bogdan Dobrelya |
fuel: status |
Fix Released |
Confirmed |
|
| 2015-12-31 09:17:29 |
Bogdan Dobrelya |
fuel: importance |
High |
Critical |
|
| 2015-12-31 09:17:32 |
Bogdan Dobrelya |
fuel/7.0.x: status |
Fix Released |
Confirmed |
|
| 2015-12-31 09:17:34 |
Bogdan Dobrelya |
fuel/8.0.x: status |
Fix Released |
Confirmed |
|
| 2015-12-31 09:17:43 |
Bogdan Dobrelya |
fuel/7.0.x: assignee |
Sergii Golovatiuk (sgolovatiuk) |
Fuel Library Team (fuel-library) |
|
| 2015-12-31 09:17:52 |
Bogdan Dobrelya |
bug task deleted |
fuel/8.0.x |
|
|
| 2015-12-31 09:33:26 |
Bogdan Dobrelya |
fuel: status |
Confirmed |
In Progress |
|
| 2015-12-31 09:35:11 |
Bogdan Dobrelya |
tags |
area-library pacemaker rabbitmq rca-done |
area-library ha pacemaker rabbitmq tricky |
|
| 2016-01-04 16:41:41 |
Bogdan Dobrelya |
fuel: status |
In Progress |
Triaged |
|
| 2016-01-05 11:22:26 |
Bogdan Dobrelya |
bug task added |
|
mos |
|
| 2016-01-05 11:22:39 |
Bogdan Dobrelya |
mos: assignee |
|
MOS Packaging Team (mos-packaging) |
|
| 2016-01-05 11:22:44 |
Bogdan Dobrelya |
mos: status |
New |
Triaged |
|
| 2016-01-05 11:22:49 |
Bogdan Dobrelya |
fuel: status |
Triaged |
In Progress |
|
| 2016-01-05 11:22:53 |
Bogdan Dobrelya |
mos: milestone |
|
8.0 |
|
| 2016-01-05 11:22:57 |
Bogdan Dobrelya |
mos: importance |
Undecided |
Critical |
|
| 2016-01-08 10:48:05 |
Bogdan Dobrelya |
fuel/7.0.x: status |
Confirmed |
Invalid |
|
| 2016-01-08 10:48:09 |
Bogdan Dobrelya |
fuel: status |
In Progress |
Invalid |
|
| 2016-01-08 11:05:41 |
Ivan Udovichenko |
mos: status |
Triaged |
In Progress |
|
| 2016-01-12 14:08:02 |
Igor Yozhikov |
mos: assignee |
MOS Packaging Team (mos-packaging) |
Ivan Udovichenko (iudovichenko) |
|
| 2016-01-13 14:24:14 |
Igor Yozhikov |
mos: status |
In Progress |
Fix Committed |
|
| 2016-01-13 17:42:56 |
Nastya Urlapova |
mos: status |
Fix Committed |
Confirmed |
|
| 2016-01-14 09:59:00 |
Bogdan Dobrelya |
nominated for series |
|
fuel/mitaka |
|
| 2016-01-14 09:59:00 |
Bogdan Dobrelya |
bug task added |
|
fuel/mitaka |
|
| 2016-01-14 09:59:12 |
Bogdan Dobrelya |
fuel/mitaka: status |
Invalid |
Fix Committed |
|
| 2016-01-14 09:59:19 |
Bogdan Dobrelya |
fuel/mitaka: milestone |
8.0 |
9.0 |
|
| 2016-01-14 09:59:47 |
Bogdan Dobrelya |
fuel/7.0.x: status |
Invalid |
Triaged |
|
| 2016-01-14 09:59:55 |
Bogdan Dobrelya |
fuel/7.0.x: assignee |
Fuel Library Team (fuel-library) |
MOS Maintenance (mos-maintenance) |
|
| 2016-01-14 10:00:00 |
Bogdan Dobrelya |
nominated for series |
|
fuel/8.0.x |
|
| 2016-01-14 10:00:00 |
Bogdan Dobrelya |
bug task added |
|
fuel/8.0.x |
|
| 2016-01-14 10:00:12 |
Bogdan Dobrelya |
fuel/8.0.x: status |
New |
In Progress |
|
| 2016-01-14 10:00:17 |
Bogdan Dobrelya |
fuel/8.0.x: importance |
Undecided |
Critical |
|
| 2016-01-14 10:00:21 |
Bogdan Dobrelya |
fuel/8.0.x: assignee |
|
Bogdan Dobrelya (bogdando) |
|
| 2016-01-14 10:00:27 |
Bogdan Dobrelya |
fuel/8.0.x: milestone |
|
8.0 |
|
| 2016-01-14 10:00:30 |
Bogdan Dobrelya |
fuel/8.0.x: status |
In Progress |
Fix Committed |
|
| 2016-01-14 10:01:52 |
Bogdan Dobrelya |
fuel/7.0.x: assignee |
MOS Maintenance (mos-maintenance) |
Bogdan Dobrelya (bogdando) |
|
| 2016-01-14 10:01:55 |
Bogdan Dobrelya |
fuel/7.0.x: status |
Triaged |
Fix Committed |
|
| 2016-01-14 10:02:09 |
Bogdan Dobrelya |
fuel/7.0.x: status |
Fix Committed |
Triaged |
|
| 2016-01-14 10:02:13 |
Bogdan Dobrelya |
fuel/7.0.x: milestone |
7.0 |
6.1-updates |
|
| 2016-01-14 10:02:18 |
Bogdan Dobrelya |
fuel/7.0.x: milestone |
6.1-updates |
7.0-updates |
|
| 2016-01-14 10:02:23 |
Bogdan Dobrelya |
fuel/7.0.x: assignee |
Bogdan Dobrelya (bogdando) |
MOS Maintenance (mos-maintenance) |
|
| 2016-01-15 09:02:58 |
Nastya Urlapova |
mos: status |
Confirmed |
Fix Committed |
|
| 2016-01-20 11:18:14 |
Alexander Zatserklyany |
tags |
area-library ha pacemaker rabbitmq tricky |
area-library ha on-verification pacemaker rabbitmq tricky |
|
| 2016-01-21 13:15:22 |
Alexander Zatserklyany |
fuel/8.0.x: status |
Fix Committed |
Fix Released |
|
| 2016-01-21 14:19:48 |
Fuel Devops McRobotson |
bug task deleted |
mos |
|
|
| 2016-01-29 07:37:38 |
Denis Puchkin |
fuel/7.0.x: assignee |
MOS Maintenance (mos-maintenance) |
Denis Puchkin (dpuchkin) |
|
| 2016-02-03 12:32:01 |
Bogdan Dobrelya |
fuel/mitaka: status |
Fix Committed |
In Progress |
|
| 2016-02-03 12:32:04 |
Bogdan Dobrelya |
fuel/8.0.x: status |
Fix Released |
In Progress |
|
| 2016-02-03 17:38:27 |
Sergii Golovatiuk |
fuel/7.0.x: importance |
Critical |
High |
|
| 2016-02-03 17:38:28 |
Sergii Golovatiuk |
fuel/8.0.x: importance |
Critical |
High |
|
| 2016-02-03 17:38:32 |
Sergii Golovatiuk |
fuel/mitaka: importance |
Critical |
High |
|
| 2016-02-04 09:20:48 |
Bogdan Dobrelya |
fuel/mitaka: status |
In Progress |
Fix Committed |
|
| 2016-02-04 09:20:50 |
Bogdan Dobrelya |
fuel/8.0.x: status |
In Progress |
Fix Released |
|
| 2016-06-16 10:42:58 |
Mikhail Samoylov |
fuel/mitaka: status |
Fix Committed |
Fix Released |
|
| 2016-09-28 09:48:42 |
Alexey Galkin |
fuel: status |
Fix Committed |
Fix Released |
|
| 2016-10-14 07:25:12 |
Denis Puchkin |
fuel/7.0.x: status |
Triaged |
Won't Fix |
|
