From the logs #20, I could see query for /analytics/uves/virtual-machine-interface/default-domain:*:*?flat only in openc52. The query was made @ 2017-09-15 12:58:18, 2017-09-15 13:14:25 and 2017-09-15 13:30:51
During this period, I analytics-api was not not able to connect to the redis @ 172.23.10.194
The following Exception was continuously logged in the contrail-analytics-api.log
09/15/2017 01:14:18 PM [contrail-analytics-api]: Exception ConnectionError in uve cache proc. Arguments:
(u'Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.',) : traceback Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 191, in get_cache_uve
tfilter, ackfilter, False)
File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 126, in _get_uve_content
pperes = ppe.execute()
File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2000, in execute
return execute(conn, stack, raise_on_error)
File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1869, in _execute_transaction
connection.send_packed_command(all_cmds)
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 292, in send_packed_command
self.connect()
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 235, in connect
raise ConnectionError(self._error_message(e))
ConnectionError: Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.
So, either there was some connectivity issue to 172.23.10.194 or redis server couldn't accept new connections due to max client connections reached. I had seen the latter case when haproxy continuously connects to redis. Can you please check if the following line is present in haproxy config? If so, please remove this line and restart haproxy and redis service - https://bugs.launchpad.net/juniperopenstack/+bug/1648601
From the logs #20, I could see query for /analytics/ uves/virtual- machine- interface/ default- domain: *:*?flat only in openc52. The query was made @ 2017-09-15 12:58:18, 2017-09-15 13:14:25 and 2017-09-15 13:30:51
During this period, I analytics-api was not not able to connect to the redis @ 172.23.10.194
The following Exception was continuously logged in the contrail- analytics- api.log
09/15/2017 01:14:18 PM [contrail- analytics- api]: Exception ConnectionError in uve cache proc. Arguments: python2. 7/dist- packages/ opserver/ partition_ handler. py", line 191, in get_cache_uve python2. 7/dist- packages/ opserver/ partition_ handler. py", line 126, in _get_uve_content python2. 7/dist- packages/ redis/client. py", line 2000, in execute python2. 7/dist- packages/ redis/client. py", line 1869, in _execute_ transaction send_packed_ command( all_cmds) python2. 7/dist- packages/ redis/connectio n.py", line 292, in send_packed_command python2. 7/dist- packages/ redis/connectio n.py", line 235, in connect (self._ error_message( e))
(u'Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.',) : traceback Traceback (most recent call last):
File "/usr/lib/
tfilter, ackfilter, False)
File "/usr/lib/
pperes = ppe.execute()
File "/usr/lib/
return execute(conn, stack, raise_on_error)
File "/usr/lib/
connection.
File "/usr/lib/
self.connect()
File "/usr/lib/
raise ConnectionError
ConnectionError: Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.
So, either there was some connectivity issue to 172.23.10.194 or redis server couldn't accept new connections due to max client connections reached. I had seen the latter case when haproxy continuously connects to redis. Can you please check if the following line is present in haproxy config? If so, please remove this line and restart haproxy and redis service - https:/ /bugs.launchpad .net/juniperope nstack/ +bug/1648601
tcp-check connect port 6379
openc52- contrail- analytics- api logs: ------- ------- ------- -------
-------
bash-3.2$ grep -r "virtual- machine- interface" * contrail- analytics- api-stdout. log.6:172. 23.10.197 - - [2017-09-15 12:58:18] "GET /analytics/ uves/virtual- machine- interface/ default- domain: *:*?flat HTTP/1.1" 200 36103651 69.184298 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 13:14:25] "GET /analytics/ uves/virtual- machine- interface/ default- domain: *:*?flat HTTP/1.1" 200 35523881 72.361021 contrail- analytics- api-stdout. log.6:172. 23.10.194 - - [2017-09-15 13:30:51] "GET /analytics/ uves/virtual- machine- interface/ default- domain: *:*?flat HTTP/1.1" 200 35710910 72.576068 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:57:09] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 011-pr- 0949:commonmax- 011-vmi- 0949001005B- 3?flat HTTP/1.1" 200 646 0.002621 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:57:26] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 001-pr- 0093:commonmax- 001-vmi- 0093002003B- 6?flat HTTP/1.1" 200 642 0.002275 contrail- analytics- api-stdout. log.7:172. 23.10.205 - - [2017-09-15 09:31:36] "GET /virtual- machine- interfaces HTTP/1.1" 404 163 0.000527
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
openc15- contrail- analytics- api logs: ------- ------- ------- ------- machine- interface" * contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 11:21:34] "GET /analytics/ uves/virtual- machine- interfaces HTTP/1.1" 200 13762421 6.256814 contrail- analytics- api-stdout. log.6:172. 23.10.194 - - [2017-09-15 11:25:44] "GET /analytics/ uves/virtual- machine- interfaces HTTP/1.1" 200 13762421 5.524747 contrail- analytics- api-stdout. log.6:172. 23.10.197 - - [2017-09-15 11:30:18] "GET /analytics/ uves/virtual- machine- interfaces HTTP/1.1" 200 13703344 6.340615 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:38:58] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 011-pr- 0949:commonmax- 011-vmi- 0949001005B- 3?flat HTTP/1.1" 200 646 0.004041 contrail- analytics- api-stdout. log.6:172. 23.10.197 - - [2017-09-15 16:43:07] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 007-pr- 0093:commonmax- 007-vmi- 0093010003B- 3?flat HTTP/1.1" 200 643 0.003218 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:57:16] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 007-pr- 0093:commonmax- 007-vmi- 0093010003B- 3?flat HTTP/1.1" 200 643 0.003299 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:57:29] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 010-pr- 0246:commonmax- 010-vmi- 0246001004B- f?flat HTTP/1.1" 200 646 0.003203
-------
bash-3.2$ grep -r "virtual-
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
openc51- contrail- analytics- api logs: ------- ------- ------- ------- machine- interface" * contrail- analytics- api-stdout. log.6:172. 23.10.205 - - [2017-09-15 09:30:49] "GET /virtual- machine- interfaces HTTP/1.1" 404 163 0.001093 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 10:14:42] "GET /analytics/ uves/virtual- machine- interfaces HTTP/1.1" 200 13762421 3.560376 contrail- analytics- api-stdout. log.6:172. 23.10.194 - - [2017-09-15 10:19:30] "GET /analytics/ uves/virtual- machine- interfaces HTTP/1.1" 200 13762421 3.829684 contrail- analytics- api-stdout. log.6:172. 23.10.197 - - [2017-09-15 10:25:14] "GET /analytics/ uves/virtual- machine- interfaces HTTP/1.1" 200 13703344 68.426998 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:39:04] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 011-pr- 0949:commonmax- 011-vmi- 0949001005B- 3?flat HTTP/1.1" 200 646 0.002773 contrail- analytics- api-stdout. log.6:172. 23.10.197 - - [2017-09-15 16:43:56] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 007-pr- 0093:commonmax- 007-vmi- 0093010003B- 3?flat HTTP/1.1" 200 643 0.002697 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:57:22] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 007-pr- 0093:commonmax- 007-vmi- 0093010003B- 3?flat HTTP/1.1" 200 643 0.002605 contrail- analytics- api-stdout. log.6:172. 23.10.193 - - [2017-09-15 16:57:33] "GET /analytics/ uves/virtual- machine- interface/ default- domain: commonmax- 011-pr- 0414:commonmax- 011-vmi- 0414001003A- e?flat HTTP/1.1" 200 646 0.002835
-------
bash-3.2$ grep -r "virtual-
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/
log/contrail/