We have problems with oslo.messaging in our tests after restart nodes
Steps:
1. Create next cluster - Ubuntu, Simple, Flat nova-network, Cinder for volumes, Ceph for images, 1 controller, 1 compute, 2 cinder+ceph nodes
2. Deploy cluster
3. Reboot all nodes one by one
4. Open Health Check tab
5. Run tests
Expected - tests passed
Actual - tests on volume and instance creation failed. Lots of errors on compute node in nova logs:
2014-09-15 13:08:11 ERROR
oslo.messaging._drivers.impl_rabbit [req-c4c51ee4-f9b7-4e9e-a949-e694f3b21757 ] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
And on cinder node in cinder logs
2014-09-15 07:40:19 ERROR
oslo.messaging._drivers.impl_rabbit [req-2fd99d10-2de0-478e-ab3e-cabc52935455 - - - - -] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
{
"build_id": "2014-09- 15_00-01- 46", 7a55cc2c09d0790 72a41beb346" , required" : true, 1b26f4479be6be7 313beefb1c8" , 73d3f42c0009844 23b59b2312b" , 22ef9ab2af26da5 ffbfbf24b13" , groups" : [
"experimental" versions" : {
"2014. 1.1-5.1" : {
"VERSION" : {
"build_ id": "2014-09- 15_00-01- 46",
"ostf_ sha": "64cb59c681658a 7a55cc2c09d0790 72a41beb346" ,
"build_ number" : "8",
"api" : "1.0",
"nailgun_ sha": "b8d8189cc37d6d 1b26f4479be6be7 313beefb1c8" ,
"production" : "docker",
"fuelmain_ sha": "d7ed7973034bde 73d3f42c0009844 23b59b2312b" ,
"astute_ sha": "f5fbd89d1e0e1f 22ef9ab2af26da5 ffbfbf24b13" ,
"feature_ groups" : [
" experimental"
"release" : "5.1",
"fuellib_ sha": "395fd9d20a0036 03cc9ad26e16cb1 3c1c45e24e6" 03cc9ad26e16cb1 3c1c45e24e6"
"ostf_sha": "64cb59c681658a
"build_number": "8",
"auth_
"api": "1.0",
"nailgun_sha": "b8d8189cc37d6d
"production": "docker",
"fuelmain_sha": "d7ed7973034bde
"astute_sha": "f5fbd89d1e0e1f
"feature_
],
"release": "5.1",
"release_
],
}
}
},
"fuellib_sha": "395fd9d20a0036
}
We have problems with oslo.messaging in our tests after restart nodes
Steps:
1. Create next cluster - Ubuntu, Simple, Flat nova-network, Cinder for volumes, Ceph for images, 1 controller, 1 compute, 2 cinder+ceph nodes
2. Deploy cluster
3. Reboot all nodes one by one
4. Open Health Check tab
5. Run tests
Expected - tests passed
Actual - tests on volume and instance creation failed. Lots of errors on compute node in nova logs:
2014-09-15 13:08:11 ERROR
oslo.messaging. _drivers. impl_rabbit [req-c4c51ee4- f9b7-4e9e- a949-e694f3b217 57 ] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
And on cinder node in cinder logs
2014-09-15 07:40:19 ERROR
oslo.messaging. _drivers. impl_rabbit [req-2fd99d10- 2de0-478e- ab3e-cabc529354 55 - - - - -] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
Logs are attached