Mirantis OpenStack

Rabbitmq-server-murano refuses connections from instance with Murano-agent

Bug #1405702 reported by Victor Ryzhenkin
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Won't Fix
High
MOS Murano
Mirantis OpenStack 6.0-updates
6.1.x
Won't Fix
High
MOS Murano
Mirantis OpenStack 6.1

Bug Description

ISO: {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

STEPS TO REPRODUCE:
1. Deploy OpenStack Ubuntu+Neutron VLAN, Ceilometer, Mongo, Ceph volumes, Ceph images, Ceph ephemeral volumes, Swift RadosGW HA_mode (3 controllers + 1 compute)
2. Create Murano environment.
3. Add any application
4. Send environment to deploy.

Actual result:
Deploy hangs.
In murano-agent.log on instance we can see tracebacks:

2014-12-25 23:56:04.975 1356 WARNING muranoagent.app [-] Communication error
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app Traceback (most recent call last):
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/app.py",
line 125, in _wait_plan
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app with self._create_rmq_client() as mq:
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/common/messaging/mqclient.py", line 52, in __enter__
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app self.connect()
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/common/messaging/mqclient.py", line 60, in connect
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app self._connection.connect()
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 237, in connect
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app return self.connection
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app self._connection = self._establish_connection()
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app conn = self.transport.establish_connection()
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 112, in establish_connection
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app conn = self.Connection(**opts)
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 171, in __init__
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app (10, 10), # start
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app self.channel_id, allowed_methods)
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 240, in _wait_method
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app self.method_reader.read_method()
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app raise m
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app OverflowError: signed integer is greater than maximum
2014-12-25 23:56:04.975 1356 TRACE muranoagent.app
2014-12-25 23:56:14.993 1356 WARNING muranoagent.app [-] Communication error
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app Traceback (most recent call last):
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/app.py", line 125, in _wait_plan
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app with self._create_rmq_client() as mq:
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/common/messaging/mqclient.py", line 52, in __enter__
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app self.connect()
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/common/messaging/mqclient.py", line 60, in connect
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app self._connection.connect()
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 237, in connect
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app return self.connection
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app self._connection = self._establish_connection()
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app conn = self.transport.establish_connection()
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 112, in establish_connection
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app conn = self.Connection(**opts)
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 165, in __init__
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app self.transport = self.Transport(host, connect_timeout, ssl)
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 186, in Transport
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app return create_transport(host, connect_timeout, ssl)
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 299, in create_transport
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app return TCPTransport(host, connect_timeout)
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 95, in __init__
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app raise socket.error(last_err)
2014-12-25 23:56:14.993 1356 TRACE muranoagent.app error: timed out
2014-12-25 23:58:02.636 1356 TRACE muranoagent.app
2014-12-25 23:59:18.767 1356 WARNING muranoagent.app [-] Communication error
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app Traceback (most recent call last):
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/app.py",
line 125, in _wait_plan
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app with self._create_rmq_client() as mq:
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/common/me
ssaging/mqclient.py", line 52, in __enter__
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app self.connect()
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/muranoagent/common/me
ssaging/mqclient.py", line 60, in connect
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app self._connection.connect()
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py",
 line 237, in connect
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app return self.connection
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py",
 line 741, in connection
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app self._connection = self._establish_connection()
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py",
 line 696, in _establish_connection
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app conn = self.transport.establish_connection()
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamq
p.py", line 112, in establish_connection
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app conn = self.Connection(**opts)
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py",
line 180, in __init__
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app (10, 30), # tune
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel
.py", line 67, in wait
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app self.channel_id, allowed_methods)
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py",
line 240, in _wait_method
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app self.method_reader.read_method()
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.p
y", line 189, in read_method
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app raise m
2014-12-25 23:59:18.767 1356 TRACE muranoagent.app IOError: Socket closed

NOTES:
Using nmap from instance we can see, that needed port(55572) on HA proxy has status "filtered"

ec2-user@murano-trntsi44sf5er2-azropi44sf2d61-zum4ealzjqxr:~$ nmap -A -T5 -Pn 172.18.92.84 -p55572

Starting Nmap 6.40 ( http://nmap.org ) at 2014-12-26 00:30 UTC
Nmap scan report for 172.18.92.84
Host is up.
PORT STATE SERVICE VERSION
55572/tcp filtered unknown

But if we try to probe this port from controller-nodes/master-nodes, it will have status "open"

root@node-2:~# nmap -Pn 172.18.92.84 -p55572

Starting Nmap 6.40 ( http://nmap.org ) at 2014-12-26 00:40 UTC
Nmap scan report for 172.18.92.84
Host is up (0.047s latency).
PORT STATE SERVICE
55572/tcp open unknown
MAC Address: B6:79:A6:2D:F8:4C (Unknown)

Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :
Victor Ryzhenkin (vryzhenkin)
Changed in mos:
assignee: nobody → MOS Murano (mos-murano)
Timur Nurlygayanov (tnurlygayanov)
Changed in mos:
milestone: 6.0 → 6.0.1
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

This problem appears when we using virtual controllers with network driver 'virtio'. The bug doesn't reprodusible when we changed 'virtio' to 'rtl8139'.

Changed in mos:
assignee: MOS Murano (mos-murano) → Fuel Documentation Team (fuel-docs)
tags: added: release-notes
Serg Melikyan (smelikyan)
Changed in mos:
status: Confirmed → Won't Fix
Serg Melikyan (smelikyan)
tags: removed: release-notes
Changed in mos:
assignee: Fuel Documentation Team (fuel-docs) → nobody
Dmitry Mescheryakov (dmitrymex)
Changed in mos:
assignee: nobody → MOS Murano (mos-murano)
Serg Melikyan (smelikyan)
tags: added: release-notes
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Release-notes: If we host controllers at virtual machines with Network Device-model 'virtio', the networking is very unstable (significant count of connection fails). This bug is reproducible in KVM or QEMU hypervisors.
Workaround: Need to specify diferent Device-model for virtual machines. For example, with Device-model 'rtl8139', network works correctly.

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

I don't think this bug should be in the release notes, see:
https://review.openstack.org/#/c/144684/2/pages/release-notes/v6-0/1050-network.rst

Revision history for this message
Sree Hari Krishna (srhrkrishna) wrote :

Me too facing the same issue on Mirantis Dev Openstack express.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.