Rabbit mq connection closed after concurrent request

[Ashish Kumar Gupta]

Code base: stable/newton with devstack
Plugin : NSX
Version of OpenSSL and Rabbitmqserver:
root@controller:/var/log/rabbitmq# dpkg -l | grep openssl
ii libgnutls-openssl27:amd64 2.12.23-12ubuntu2.2 amd64 GNU TLS library - OpenSSL wrapper
ii openssl 1.0.1f-1ubuntu2.22 amd64 Secure Sockets Layer toolkit - cryptographic utility
ii python-openssl 0.13-2ubuntu6 amd64 Python 2 wrapper around the OpenSSL library
root@controller:/var/log/rabbitmq#

root@controller:/var/log/rabbitmq# dpkg -l | grep rabbit
ii librabbitmq1 0.4.1-1 amd64 AMQP client library written in C
ii python-librabbitmq 1.0.3-0ubuntu1 amd64 AMQP Client using the rabbitmq-c library
ii rabbitmq-server 3.2.4-1 all AMQP server written in Erlang

Try to make concurrent request to the neutron server using rally :
{
      "runner": {
        "type": "constant",
        "concurrency": 10,
        "times": 20
      },

Fails to create some neutron security group due to below error:
Rabbitmq connection getting reset:

2017-03-30 05:21:39.416 26839 ERROR vmware_nsx.plugins.nsx_v.plugin raise SSLError(e, request=request)
2017-03-30 05:21:39.416 26839 ERROR vmware_nsx.plugins.nsx_v.plugin SSLError: bad handshake: SysCallError(-1, 'Unexpected EOF')
2017-03-30 05:21:39.416 26839 ERROR vmware_nsx.plugins.nsx_v.plugin
2017-03-30 05:21:39.426 26839 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2017-03-30 05:21:39.433 26839 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2017-03-30 05:21:39.439 26839 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2017-03-30 05:21:39.550 26838 INFO neutron.wsgi [req-c7b23b88-60f3-4a2a-8833-605759c12ce3 c_rally_da68736d_p8i9325S -] 60.0.26.100 - - [30/Mar/2017 05:21:39] "GET /v2.0/networks.json?fields=id&name=external2 HTTP/1.1" 200 260 204.329335
2017-03-30 05:22:30.395 26837 ERROR oslo.messaging._drivers.impl_rabbit [-] [7984cafb-7593-4596-999b-d544c54579d4] AMQP server on 60.0.26.100:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)

Attaching the log of neutron server.

[Sean Dague (sdague)]

Automatically discovered version newton in description. If this is incorrect, please update the description to include 'https://api.launchpad.net/1.0/devstack version: ...'

[Sean Dague (sdague)]

Automatically discovered version newton in description. If this is incorrect, please update the description to include 'devstack version: ...'

[Aizuddin Zali (mymzbe)]

Do you have the RabbitMQ log? The problem was that the connection reset by the RabbitMQ.

[Launchpad Janitor (janitor)]

[Expired for devstack because there has been no activity for 60 days.]