Steps to reproduce:
1. Deploy env with Ubunu and neutron
2. When cluster is ready, run ostf
3. Destroy primary controller: virsh destroy domain_name_here
4. Wait until controller appears to offline in nailgun near 5 minutes
5. Wait the pacemaker react to changes in online nodes near 1 minutes
6. Run ostf ha suit, if it fails wait 5 minutes and run again
Expected:
all ostf tests are passed
Actual:
Ha for rabbit failed:
Can not get rabbit channel list in 20 second. Please refer to OpenStack logs for more details on node-4:
uel_health.ha_base: DEBUG: Output is resource master_p_rabbitmq-server is running on: node-5.test.domain.local Master
resource master_p_rabbitmq-server is running on: node-4.test.domain.local
resource master_p_rabbitmq-server is NOT running
Command 'rabbitmqctl list_channels', exit status: 2, Error:
Listing channels ...
Error: unable to connect to node 'rabbit@node-4': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit@node-4']
rabbit@node-4:
* connected to epmd (port 4369) on node-4
* node rabbit@node-4 up, 'rabbit' application running
current node details:
- node name: 'rabbitmqctl7872@node-4'
- home dir: /var/lib/rabbitmq
- cookie hash: soeIWU2jk2YNseTyDSlsEA==
On node-4 in rabbit.logs:
2015-07-08T01:14:53.340302+00:00 err: Killed -> in start-error log
2015-07-08T01:14:33.321304+00:00 notice: =CRASH REPORT==== 8-Jul-2015::01:14:32 ===
2015-07-08T01:14:33.321304+00:00 notice: crasher:
2015-07-08T01:14:33.321304+00:00 notice: initial call: rabbit_reader:init/2
2015-07-08T01:14:33.321304+00:00 notice: pid: <0.2773.0>
2015-07-08T01:14:33.321304+00:00 notice: registered_name: []
2015-07-08T01:14:33.321304+00:00 notice: exception exit: channel_termination_timeout
2015-07-08T01:14:33.321304+00:00 notice: in function rabbit_reader:wait_for_channel_termination/3 (src/rabbit_reader.erl, line 568)
2015-07-08T01:14:33.321304+00:00 notice: in call from rabbit_reader:handle_exception/3 (src/rabbit_reader.erl, line 596)
2015-07-08T01:14:33.321304+00:00 notice: in call from rabbit_reader:terminate/2 (src/rabbit_reader.erl, line 433)
2015-07-08T01:14:33.321304+00:00 notice: in call from rabbit_reader:handle_other/2 (src/rabbit_reader.erl, line 360)
2015-07-08T01:14:33.321304+00:00 notice: in call from rabbit_reader:mainloop/4 (src/rabbit_reader.erl, line 330)
2015-07-08T01:14:33.321304+00:00 notice: in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 281)
2015-07-08T01:14:33.321304+00:00 notice: in call from rabbit_reader:start_connection/5 (src/rabbit_reader.erl, line 255)
2015-07-08T01:14:33.321304+00:00 notice: ancestors: [<0.2771.0>,rabbit_tcp_client_sup,rabbit_sup,<0.627.0>]
2015-07-08T01:14:33.321304+00:00 notice: messages: [{'EXIT',#Port<0.3401>,normal}]
2015-07-08T01:14:33.321304+00:00 notice: links: []
2015-07-08T01:14:33.321304+00:00 notice: dictionary: [{{channel,1},
2015-07-08T01:14:33.321304+00:00 notice: {<0.2781.0>,{method,rabbit_framing_amqp_0_9_1}}},
2015-07-08T01:14:33.321304+00:00 notice: {{ch_pid,<0.2781.0>},{1,#Ref<0.0.0.16377>}},
2015-07-08T01:14:33.321304+00:00 notice: {process_name,
2015-07-08T01:14:33.321304+00:00 notice: {rabbit_reader,
2015-07-08T01:14:33.321304+00:00 notice: <<"127.0.0.1:37213 -> 127.0.0.1:5673">>}}]
2015-07-08T01:14:33.321304+00:00 notice: trap_exit: true
2015-07-08T01:14:33.321304+00:00 notice: status: running
2015-07-08T01:14:33.321304+00:00 notice: heap_size: 987
2015-07-08T01:14:33.321304+00:00 notice: stack_size: 27
2015-07-08T01:14:33.321304+00:00 notice: reductions: 5590
2015-07-08T01:14:33.321304+00:00 notice: neighbours:
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "7.0"
openstack_version: "2014.2.2-7.0"
api: "1.0"
build_number: "239"
build_id: "2015-07-07_18-02-36"
nailgun_sha: "2f8d9be6bca0c32a83acda02e1689a44f0d8eefe"
python-fuelclient_sha: "018c53561baa87397e50368c7f48674829c322c0"
astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
fuel-library_sha: "93a5fe7251f3eb932d588ba338920e9e7cfe7f0c"
fuel-ostf_sha: "ba48d52474515bcd1ad2a9dfa5228bff36ee98d7"
fuelmain_sha: "c53bb47881f0bb9ba14274a596923d9860140406"
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.ha_neutron_destructive/19/#showFailuresLink
