| 2016-03-25 08:36:26 |
Alexander Koryagin |
bug |
|
|
added bug |
| 2016-03-25 08:38:16 |
Alexander Koryagin |
description |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions:
1) OK - Install oslo.messaging-check-tool.git
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool.git
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
|
| 2016-03-25 09:01:30 |
Alexander Koryagin |
description |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool.git
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool.git
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
|
| 2016-03-25 09:02:04 |
Alexander Koryagin |
description |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool.git
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool on first controller:
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
|
| 2016-03-25 09:03:26 |
Alexander Koryagin |
description |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool on first controller:
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool on first controller:
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
|
| 2016-03-25 09:47:44 |
Alexander Koryagin |
description |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool on first controller:
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000 |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool on first controller:
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
root@node-2:~# rabbitmqctl list_policies
Listing policies ...
/ ha-notif all ^(event|metering|notifications)\\. {"ha-mode":"all","ha-sync-mode":"automatic"} 0
/ heat_rpc_expire all ^heat-engine-listener\\. {"expires":3600000} 1
/ tasks_expire all ^tasks\\. {"expires":3600000} 1
/ results_expire all ^results\\. {"expires":3600000} 1 |
|
| 2016-03-25 13:17:17 |
Dina Belova |
mos: status |
New |
Invalid |
|
| 2016-03-25 15:11:17 |
Alexander Koryagin |
mos: status |
Invalid |
New |
|
| 2016-03-28 12:05:18 |
Alexander Koryagin |
description |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool on first controller:
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
root@node-2:~# rabbitmqctl list_policies
Listing policies ...
/ ha-notif all ^(event|metering|notifications)\\. {"ha-mode":"all","ha-sync-mode":"automatic"} 0
/ heat_rpc_expire all ^heat-engine-listener\\. {"expires":3600000} 1
/ tasks_expire all ^tasks\\. {"expires":3600000} 1
/ results_expire all ^results\\. {"expires":3600000} 1 |
Hello,
Please take a look at the following issue:
After restart of 'rabbitmq-server' service on all controllers, messages, generated with 'oslo.messaging-check-tool' were lost.
Note: I am not restarting service on ALL controllers at ONCE.
My actions are:
- Restart rabbit service on one controller;
- Wait till it'll be up and running;
- Wait till this controller will present in cluster;
- Perform the same actions on next controller.
My env is MOS 9.0 (ISO: fuel-9.0-94-2016-03-21_14-00-00.iso)
With 3x controller, 1x compute-cinder.
Actions performed from controller(s):
1) OK - Install oslo.messaging-check-tool on first controller:
# apt-get update
# apt-get install git python-pip python-dev -y
# cd /root/
# git clone https://github.com/dmitrymex/oslo.messaging-check-tool.git
# cd /root/oslo.messaging-check-tool/
# pip install -r requirements.txt -r test-requirements.txt
# dpkg -i oslo.messaging-check-tool_1.0-1~u14.04+mos1_all.deb
# apt-get -f install -y
2) OK - Get nodes inside RabbitMQ cluster:
# rabbitmqctl cluster_status | grep -A1 'running_nodes'
{running_nodes,['rabbit@messaging-node-1','rabbit@messaging-node-4',
'rabbit@messaging-node-2']},
3) OK - Get IPs of nodes:
# getent hosts node-1 --> 10.109.1.4 node-1.test.domain.local node-1
# getent hosts node-2 --> 10.109.1.7 node-2.test.domain.local node-2
# getent hosts node-4 --> 10.109.1.6 node-4.test.domain.local node-4
4) OK - Fill oslo config file:
# cat /root/oslo.messaging-check-tool/oslo_msg_check.conf
[DEFAULT]
debug=true
[oslo_messaging_rabbit]
rabbit_hosts = 10.109.1.4:5673, 10.109.1.6:5673, 10.109.1.7:5673
rabbit_userid = nova
rabbit_password = Ajl9OxOMW2mgB7fZRZbH6aPu
5) OK - Generate and consume 10000 messages without rabbitmq-server restart:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) OK - Generate and consume 10000 messages WITH rabbitmq-server restart on ONE controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> OK - Consumed 10000 messages
6) NOK - Generate and consume 10000 messages WITH rabbitmq-server restart on ALL controller:
On first controller:
# cd /root/oslo.messaging-check-tool/
# oslo_msg_load_generator --config-file oslo_msg_check.conf --messages-to-send 10000 --nodebug
Perform restart on each controller one-by-one and check cluster status after it:
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Remember number before restart. (For me: 20)
# service rabbitmq-server restart && sleep 10
# rabbitmqctl cluster_status | grep -A1 'running_nodes' #\\ check that all 3 nodes present
# rabbitmqctl list_queues slave_pids name | grep `hostname` | wc -l #\\ Check that num is the same after restart. (For me: 20)
After that perform the same commands above on next controller.
On first controller:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
Re-try on first controller after 5 minutes:
# oslo_msg_load_consumer --config-file oslo_msg_check.conf --nodebug
>>> NOK - Consumed 0 messages \\\ Should be 10000
root@node-2:~# rabbitmqctl list_policies
Listing policies ...
/ ha-notif all ^(event|metering|notifications)\\. {"ha-mode":"all","ha-sync-mode":"automatic"} 0
/ heat_rpc_expire all ^heat-engine-listener\\. {"expires":3600000} 1
/ tasks_expire all ^tasks\\. {"expires":3600000} 1
/ results_expire all ^results\\. {"expires":3600000} 1 |
|
| 2016-03-28 13:08:24 |
Dina Belova |
mos: status |
New |
Confirmed |
|
| 2016-03-28 13:08:27 |
Dina Belova |
mos: importance |
Undecided |
High |
|
| 2016-03-28 13:08:32 |
Dina Belova |
mos: assignee |
|
MOS Oslo (mos-oslo) |
|
| 2016-03-28 13:08:34 |
Dina Belova |
mos: milestone |
|
9.0 |
|
| 2016-03-28 13:20:34 |
Bug Checker Bot |
tags |
rabbitmq |
need-info rabbitmq |
|
| 2016-04-15 13:06:55 |
Sergey Shevorakov |
tags |
need-info rabbitmq |
need-info rabbitmq swarm-blocker |
|
| 2016-04-19 09:14:38 |
Fuel Devops McRobotson |
mos/10.0.x: importance |
Undecided |
High |
|
| 2016-04-19 09:14:38 |
Fuel Devops McRobotson |
mos/10.0.x: status |
New |
Confirmed |
|
| 2016-04-19 09:14:38 |
Fuel Devops McRobotson |
mos/10.0.x: milestone |
|
10.0 |
|
| 2016-04-19 09:14:38 |
Fuel Devops McRobotson |
mos/10.0.x: assignee |
|
MOS Oslo (mos-oslo) |
|
| 2016-05-20 16:46:35 |
Dmitry Mescheryakov |
mos: assignee |
MOS Oslo (mos-oslo) |
Dmitry Mescheryakov (dmitrymex) |
|
| 2016-05-20 16:46:37 |
Dmitry Mescheryakov |
mos/10.0.x: assignee |
MOS Oslo (mos-oslo) |
Dmitry Mescheryakov (dmitrymex) |
|
| 2016-06-01 14:57:01 |
Dmitry Mescheryakov |
mos: status |
Confirmed |
In Progress |
|
| 2016-06-01 14:57:05 |
Dmitry Mescheryakov |
mos/10.0.x: status |
Confirmed |
In Progress |
|
| 2016-06-08 11:37:45 |
Dina Belova |
tags |
need-info rabbitmq swarm-blocker |
move-to-mu need-info rabbitmq swarm-blocker |
|
| 2016-06-08 11:37:51 |
Dina Belova |
mos: milestone |
9.0 |
9.0-updates |
|
| 2016-07-11 11:26:28 |
Dmitry Mescheryakov |
tags |
move-to-mu need-info rabbitmq swarm-blocker |
10.0-reviewed move-to-mu need-info rabbitmq swarm-blocker |
|
| 2016-08-16 10:50:26 |
Sergey Shevorakov |
tags |
10.0-reviewed move-to-mu need-info rabbitmq swarm-blocker |
10.0-reviewed move-to-mu need-info rabbitmq |
|
| 2016-08-19 15:11:05 |
Dmitry Mescheryakov |
mos/10.0.x: status |
In Progress |
Invalid |
|
| 2016-08-31 10:39:13 |
Dmitry Mescheryakov |
mos: status |
In Progress |
Fix Committed |
|
| 2016-09-09 11:50:03 |
Alexey Galkin |
mos: status |
Fix Committed |
Fix Released |
|
