Activity log for bug #1436343

Date Who What changed Old value New value Message
2015-03-25 13:48:42 Alexey Khivin bug added bug
2015-03-25 13:49:10 Alexey Khivin description Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill on of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ cluster within the cloud http://paste.openstack.org/show/196563/ [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions: 2014.2-6.1: VERSION: api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ cluster within the cloud http://paste.openstack.org/show/196563/ [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1'
2015-03-25 13:54:25 Alexey Khivin fuel: importance Undecided High
2015-03-25 14:52:34 Alexey Khivin description Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ cluster within the cloud http://paste.openstack.org/show/196563/ [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1' Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ cluster within the cloud http://paste.openstack.org/show/196563/ This state is _permanent_ and will not be fixed by OCF scrypt without admin actions [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1'
2015-03-25 15:10:39 Alexey Khivin summary RabbitMQ split brain RabbitMQ split-brain
2015-03-25 15:11:26 Alexey Khivin description Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ cluster within the cloud http://paste.openstack.org/show/196563/ This state is _permanent_ and will not be fixed by OCF scrypt without admin actions [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1' Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ clusters within the cloud http://paste.openstack.org/show/196563/ This state is _permanent_ and will not be fixed by OCF script without admin actions [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1'
2015-03-25 15:29:39 Alexey Khivin description Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ clusters within the cloud http://paste.openstack.org/show/196563/ This state is _permanent_ and will not be fixed by OCF script without admin actions [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1' Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ clusters within the cloud http://paste.openstack.org/show/196563/ This state is _permanent_ and will not be fixed by OCF script without admin actions node-7 and node-10 both has the master status with the slave node-8 [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1'
2015-03-25 15:31:03 Davanum Srinivas (DIMS) summary RabbitMQ split-brain Pacemaker cluster failure leads to RabbitMQ split-brain
2015-03-27 11:13:35 Bogdan Dobrelya fuel: milestone 6.1
2015-03-27 11:15:01 Bogdan Dobrelya summary Pacemaker cluster failure leads to RabbitMQ split-brain Pacemaker cluster failure leads to Corosync split-brain
2015-03-27 11:16:40 Bogdan Dobrelya fuel: status New Confirmed
2015-03-27 11:17:11 Bogdan Dobrelya summary Pacemaker cluster failure leads to Corosync split-brain Consequent -1 node failures lead to Corosync split-brain
2015-03-27 16:55:13 Alexey Khivin tags ha rabbitmq corosync ha rabbitmq
2015-03-31 10:56:08 Bogdan Dobrelya fuel: importance High Critical
2015-03-31 11:16:01 Bogdan Dobrelya fuel: importance Critical High
2015-03-31 11:28:24 Alexey Khivin description Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ clusters within the cloud http://paste.openstack.org/show/196563/ This state is _permanent_ and will not be fixed by OCF script without admin actions node-7 and node-10 both has the master status with the slave node-8 [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1' Multinode with HA with 3 controllers (node-10, node-7, node-8) + 2 Node (compute+storage) Juno on Ubuntu 14.01 Test case: 1. kill one of controllers (master controller or slave controller for rabbitmq) 2. wait until RabbitMQ cluster became reassembled by ocf script and became stable 3. turn on controller 4. wait until RabbitMQ cluster became reassembled by ocf script and became stable 5. goto 1 after maybe 5-7 cycles we see two different RabbitMQ clusters within the cloud http://paste.openstack.org/show/196563/ This state is _permanent_ and will not be fixed by OCF script without admin actions node-7 and node-10 both has the master status with the slave node-8 [root@fuel ~]# fuel --f DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient. api: '1.0' astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9 auth_required: true build_id: 2015-03-23_15-29-20 build_number: '218' feature_groups: - mirantis fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265 fuelmain_sha: a05ab877af31924585c81081f45305700961458e nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3 ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867 production: docker python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043 release: '6.1' release_versions:   2014.2-6.1:     VERSION:       api: '1.0'       astute_sha: 4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9       build_id: 2015-03-23_15-29-20       build_number: '218'       feature_groups:       - mirantis       fuellib_sha: a0265ae47bb2307a6967a3f1dd06fe222c561265       fuelmain_sha: a05ab877af31924585c81081f45305700961458e       nailgun_sha: 7c100f47450ea1a910e19fa09f78d586cb2bc0d3       ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867       production: docker       python-fuelclient_sha: 3624051242c83fdbdd1df9a0e466797c06b75043       release: '6.1'
2015-04-01 10:59:00 Bogdan Dobrelya tags corosync ha rabbitmq corosync ha rabbitmq to-be-covered-by-tests
2015-04-03 12:23:59 Vladimir Kuklin fuel: assignee Fuel Library Team (fuel-library) Bogdan Dobrelya (bogdando)
2015-04-03 13:58:15 Bogdan Dobrelya marked as duplicate 1439120
2015-04-08 19:52:50 Davanum Srinivas (DIMS) removed duplicate marker 1439120
2015-04-09 10:37:58 Bogdan Dobrelya fuel: importance High Critical
2015-04-10 15:08:58 Bogdan Dobrelya tags corosync ha rabbitmq to-be-covered-by-tests corosync ha to-be-covered-by-tests
2015-04-10 15:10:10 Bogdan Dobrelya fuel: status Confirmed In Progress
2015-04-10 15:11:01 Bogdan Dobrelya attachment added logs.tar.gz https://bugs.launchpad.net/fuel/+bug/1436343/+attachment/4371642/+files/logs.tar.gz
2015-04-10 15:54:52 Fuel Devops McRobotson fuel/6.0.x: milestone 6.0.1
2015-04-10 15:54:52 Fuel Devops McRobotson fuel/6.0.x: assignee Bogdan Dobrelya (bogdando)
2015-04-13 08:56:37 Bogdan Dobrelya fuel/6.0.x: status New Invalid
2015-04-13 10:36:10 Bogdan Dobrelya attachment added logs_minority.tar.gz https://bugs.launchpad.net/fuel/+bug/1436343/+attachment/4373771/+files/logs_minority.tar.gz
2015-04-14 07:24:43 Bogdan Dobrelya fuel: status In Progress Triaged
2015-04-14 07:52:35 Bogdan Dobrelya marked as duplicate 1443800
2015-04-15 11:13:09 Leontii Istomin tags corosync ha to-be-covered-by-tests corosync ha scale to-be-covered-by-tests