"[Errno 110] Connection timed out" nova and neutron from compute node to rabbitmq

[Leontii Istomin]

We performed shaker tests (http://pyshaker.readthedocs.org/en/latest/)

14:08:22 dense_l2.yaml
14:33:07 dense_l3_east_west.yaml
14:55:41 dense_l3_north_south.yaml
15:13:08 full_l2.yaml
15:37:31 full_l3_east_west.yaml
15:58:15 full_l3_north_south.yaml
16:23:40 perf_l2.yaml
16:27:17 perf_l3_east_west.yaml
16:30:53 perf_l3_north_south.yaml

rabbitmq hasn't been broken

from neutron-all.log from compute node-10: http://paste.openstack.org/show/352441/
from nova-all.log from compute node-10: http://paste.openstack.org/show/352442/

Check this tace on all nodes: http://paste.openstack.org/show/352667/
from rabbitmq.log on node-7: http://paste.openstack.org/show/352614/
from rabbitmq.log on node-9: http://paste.openstack.org/show/352713/
from rabbitmq.log on node-14: http://paste.openstack.org/show/352716/
from pacemaker.log: http://paste.openstack.org/show/352642/

At the time we didn't face with an issue like "instance is in ERROR state" or "neutron port hasn't been created". No issues which we seen. It means that reconnect was succesfull.

Diagnostic snapshot is here: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-07-07_16-53-20.tar.xz

[Bogdan Dobrelya (bogdando)]

Please elaborate the impact of this bug to cloud operations

[Vitaly Sedelnik (vsedelnik)]

Leontiy - please clearly specify MOS version the issue was observed in to enable confirmation of this bug, thanks!

[Leontii Istomin (listomin)]

We used 6.1 release build (6.1-525)

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'

[Timur Nurlygayanov (tnurlygayanov)]

MOS Oslo team, please, take a look.

[Timur Nurlygayanov (tnurlygayanov)]

Looks like we have some 'timour errors' in logs of OpenStack components during performance data plane testing.

[Yakov Beschasnov (yashumitsu)]

Hi there

I'm so sorry. I forgot to set up jumbo frames on a network switch and because of that we had a same timeout error (MTU is set to 9000).

[Davanum Srinivas (DIMS) (dims-v)]

The last comment indicates the problem is not with oslo.messaging, can you please let us know if it is or not?

[Roman Podoliaka (rpodolyaka)]

Incomplete for a month now, closing as Invalid