Comment 23 for bug 1701426

Revision history for this message
Sundaresan Rajangam (srajanga) wrote :

From the logs #20, I could see query for /analytics/uves/virtual-machine-interface/default-domain:*:*?flat only in openc52. The query was made @ 2017-09-15 12:58:18, 2017-09-15 13:14:25 and 2017-09-15 13:30:51
During this period, I analytics-api was not not able to connect to the redis @ 172.23.10.194

The following Exception was continuously logged in the contrail-analytics-api.log

09/15/2017 01:14:18 PM [contrail-analytics-api]: Exception ConnectionError in uve cache proc. Arguments:
(u'Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.',) : traceback Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 191, in get_cache_uve
    tfilter, ackfilter, False)
  File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 126, in _get_uve_content
    pperes = ppe.execute()
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2000, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1869, in _execute_transaction
    connection.send_packed_command(all_cmds)
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 292, in send_packed_command
    self.connect()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 235, in connect
    raise ConnectionError(self._error_message(e))
ConnectionError: Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.

So, either there was some connectivity issue to 172.23.10.194 or redis server couldn't accept new connections due to max client connections reached. I had seen the latter case when haproxy continuously connects to redis. Can you please check if the following line is present in haproxy config? If so, please remove this line and restart haproxy and redis service - https://bugs.launchpad.net/juniperopenstack/+bug/1648601

tcp-check connect port 6379

openc52-contrail-analytics-api logs:
-----------------------------------

bash-3.2$ grep -r "virtual-machine-interface" *
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.197 - - [2017-09-15 12:58:18] "GET /analytics/uves/virtual-machine-interface/default-domain:*:*?flat HTTP/1.1" 200 36103651 69.184298
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 13:14:25] "GET /analytics/uves/virtual-machine-interface/default-domain:*:*?flat HTTP/1.1" 200 35523881 72.361021
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.194 - - [2017-09-15 13:30:51] "GET /analytics/uves/virtual-machine-interface/default-domain:*:*?flat HTTP/1.1" 200 35710910 72.576068
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:09] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-011-pr-0949:commonmax-011-vmi-0949001005B-3?flat HTTP/1.1" 200 646 0.002621
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:26] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-001-pr-0093:commonmax-001-vmi-0093002003B-6?flat HTTP/1.1" 200 642 0.002275
log/contrail/contrail-analytics-api-stdout.log.7:172.23.10.205 - - [2017-09-15 09:31:36] "GET /virtual-machine-interfaces HTTP/1.1" 404 163 0.000527

openc15-contrail-analytics-api logs:
-----------------------------------
bash-3.2$ grep -r "virtual-machine-interface" *
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 11:21:34] "GET /analytics/uves/virtual-machine-interfaces HTTP/1.1" 200 13762421 6.256814
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.194 - - [2017-09-15 11:25:44] "GET /analytics/uves/virtual-machine-interfaces HTTP/1.1" 200 13762421 5.524747
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.197 - - [2017-09-15 11:30:18] "GET /analytics/uves/virtual-machine-interfaces HTTP/1.1" 200 13703344 6.340615
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:38:58] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-011-pr-0949:commonmax-011-vmi-0949001005B-3?flat HTTP/1.1" 200 646 0.004041
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.197 - - [2017-09-15 16:43:07] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-007-pr-0093:commonmax-007-vmi-0093010003B-3?flat HTTP/1.1" 200 643 0.003218
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:16] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-007-pr-0093:commonmax-007-vmi-0093010003B-3?flat HTTP/1.1" 200 643 0.003299
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:29] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-010-pr-0246:commonmax-010-vmi-0246001004B-f?flat HTTP/1.1" 200 646 0.003203

openc51-contrail-analytics-api logs:
-----------------------------------
bash-3.2$ grep -r "virtual-machine-interface" *
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.205 - - [2017-09-15 09:30:49] "GET /virtual-machine-interfaces HTTP/1.1" 404 163 0.001093
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 10:14:42] "GET /analytics/uves/virtual-machine-interfaces HTTP/1.1" 200 13762421 3.560376
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.194 - - [2017-09-15 10:19:30] "GET /analytics/uves/virtual-machine-interfaces HTTP/1.1" 200 13762421 3.829684
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.197 - - [2017-09-15 10:25:14] "GET /analytics/uves/virtual-machine-interfaces HTTP/1.1" 200 13703344 68.426998
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:39:04] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-011-pr-0949:commonmax-011-vmi-0949001005B-3?flat HTTP/1.1" 200 646 0.002773
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.197 - - [2017-09-15 16:43:56] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-007-pr-0093:commonmax-007-vmi-0093010003B-3?flat HTTP/1.1" 200 643 0.002697
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:22] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-007-pr-0093:commonmax-007-vmi-0093010003B-3?flat HTTP/1.1" 200 643 0.002605
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:33] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-011-pr-0414:commonmax-011-vmi-0414001003A-e?flat HTTP/1.1" 200 646 0.002835