Juniper Openstack

[ 3.0.2.0-35~liberty ] Redis-UVE: connection down:ConnectionError in uve stream proc

Bug #1580716 reported by chhandak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Committed
Medium
Anish Mehta
R3.0
Fix Committed
Medium
Anish Mehta
Juniper Openstack r3.0.2.0

Bug Description

Observed this log in tor scale setup. During that time contrail-analytics-api was stuck in initializing state. Recovered eventually. On the redis server couple off connection still stuck in TIME_WAIT state

Setup has following scale config:
16000 VMI
16000 LIF
8000 VN

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api initializing (Redis-UVE:172.17.90.3:6379[None] connection down)
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

root@5b7s3:~# netstat -anp | grep 6379| more
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.2:59608 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.3:45454 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56090 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:44264 172.17.90.3:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:59971 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:6379 172.17.90.2:37024 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.2:37023 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.3:44264 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.2:59607 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56608 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:43779 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:56603 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:54907 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:57009 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.3:35642 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.2:37146 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.2:33520 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56592 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:57057 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 127.0.0.1:38543 127.0.0.1:6379 ESTABLISHED 11216/python
tcp 0 0 172.17.90.3:56597 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.2:39224 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.2:59613 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:54801 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:57034 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.2:40544 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:57035 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:34212 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:56009 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:45777 172.17.90.3:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:57037 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:57062 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.2:52236 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56588 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.2:54175 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:35723 172.17.90.3:6379 ESTABLISHED 11216/python
tcp 0 0 127.0.0.1:6379 127.0.0.1:38368 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56587 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:56008 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:6379 172.17.90.2:59454 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:6379 172.17.90.2:45979 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:55974 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 127.0.0.1:6379 127.0.0.1:60402 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:55977 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:6379 172.17.90.2:54174 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:57059 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:56586 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.2:59609 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56094 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:56601 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:60597 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:57049 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.2:45974 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56605 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:6379 172.17.90.3:35722 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:57011 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:57014 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:34213 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:34210 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:6379 172.17.90.2:44046 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:56595 172.17.90.2:6379 TIME_WAIT -
tcp 0 0 172.17.90.3:35722 172.17.90.3:6379 ESTABLISHED 11216/python
tcp 0 0 172.17.90.3:54800 172.17.90.2:6379 ESTABLISHED 11215/python
tcp 0 0 172.17.90.3:6379 172.17.90.2:37147 ESTABLISHED 20940/redis-server
tcp 0 0 172.17.90.3:57047 172.17.90.2:6379 TIME_WAIT -
tcp6 0 0 :::6379 :::* LISTEN 20940/redis-server

Log from contrail-analytics-api.log
-------------------------------------
05/11/2016 11:00:00 AM [contrail-analytics-api]: Exception ConnectionError in uve stream proc. Arguments:
("Error while reading from socket: ('timed out',)",) : traceback Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 354, in _run
    for message in pb.listen():
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1715, in listen
    r = self.parse_response()
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1659, in parse_response
    response = self.connection.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 315, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 105, in read_response
    response = self.read()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 94, in read
    (e.args,))
ConnectionError: Error while reading from socket: ('timed out',)

05/11/2016 11:00:01 AM [contrail-analytics-api]: Exception ConnectionError in uve stream proc. Arguments:
("Error while reading from socket: ('timed out',)",) : traceback Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 354, in _run
    for message in pb.listen():
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1715, in listen
    r = self.parse_response()
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1659, in parse_response
    response = self.connection.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 315, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 105, in read_response
    response = self.read()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 94, in read
    (e.args,))
ConnectionError: Error while reading from socket: ('timed out',)

Role Distribution is Testbed
-------------------------------
host1 ='root@10.87.121.68'
host2 ='root@10.87.121.69'
host3 ='root@10.87.121.70'
host4 ='root@10.87.121.71'
host5 ='root@10.87.121.72'
host6 ='root@10.87.121.73'
host7 ='root@10.87.121.74'
host8 ='root@10.87.121.75'
host9 ='root@10.87.121.76'
host10 ='root@10.87.121.77'

control_data = {
    host1 : { 'ip': '172.17.90.1/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host2 : { 'ip': '172.17.90.2/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host3 : { 'ip': '172.17.90.3/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host4 : { 'ip': '172.17.90.4/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host5 : { 'ip': '172.17.90.5/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host6 : { 'ip': '172.17.90.6/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host7 : { 'ip': '172.17.90.7/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host8 : { 'ip': '172.17.90.8/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host9 : { 'ip': '172.17.90.9/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
    host10 : { 'ip': '172.17.90.10/24', 'gw' : '172.17.90.254', 'device':'p514p1' },
}

env.roledefs = {
    'all': [host1,host2,host3,host4,host5,host6,host7,host8,host9,host10],

    'cfgm': [host1,host2,host3],
    'openstack': [host1],
    'webui': [host1],
    'control': [host2,host3],
    'compute': [host4,host5,host6,host7,host8,host9,host10],
    'tsn': [host4,host5,host6,host7],
    'toragent': [host4,host5,host6,host7],
    'collector': [host2,host3],
    'database': [host1,host2,host3],
    'build': [host_build],
}

chhandak (chhandak)
Changed in juniperopenstack:
importance: Undecided → Medium
assignee: nobody → Raj Reddy (rajreddy)
summary: - Redis-UVE: connection down:ConnectionError in uve stream proc
+ [ 3.0.2.0-35~liberty ] Redis-UVE: connection down:ConnectionError in uve
+ stream proc
Revision history for this message
chhandak (chhandak) wrote :

Logs copied to /auto/cores/1580716

information type: Proprietary → Public
Raj Reddy (rajreddy)
Changed in juniperopenstack:
assignee: Raj Reddy (rajreddy) → Anish Mehta (amehta00)
Revision history for this message
Anish Mehta (amehta00) wrote :

We have decreased load on redis and added more instrumentation as part of:
https://review.opencontrail.org/20168
Please open a bug again if this issue is seen.

Changed in juniperopenstack:
status: New → Fix Committed
Revision history for this message
Anish Mehta (amehta00) wrote :

We have decreased load on redis and added more instrumentation as part of:
https://review.opencontrail.org/20169
Please open a bug again if this issue is seen.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.