Incorrect response from analytics api server when queried for number of VMIs greater than 48K

[vijaya kumar shankaran]

Customer is running a Curl against analytics api for the number of Virtual Machine interface.
When the number of VMI interface is less than 48K the response from the Analytics server matches with that of VMI on the setup. However, when there are large number of VMI >48K the response doesn’t match with the existing with VMI and we get different response each time. We get a http 200 OK response in both the case.
Following response matches with the existing VMI
Analytics API / configAPI
11 k VMI: 11135/11135 OK GET 200 5886 759 12.00 3 60 4
30 k VMI: 30000/30000 OK GET 200 15904063 35.785529
36 k VMI: 36000/36000 OK GET 200 19084807 45.094492
44.3 k VMI: 44327/44327 OK GET 200 23434148 59.721491
45 k VMI: 45000/45000 OK GET 200 23826101 62.304583

The ones that doesn’t match are
46.8 k VMI: 46060/46800 NG GET 200 24316603 59.041701
60 k VMI: 55704/60000 NG GET 200 29530343 97.429255
177 k VMI: 159975/177061 NG GET 200 84574245 323.764904

Customer is on v3.1.3 Build 75 and the curl command executed is
curl http://localhost:8081/analytics/uves/virtual-machine-interface/default-domain:*:*?flat

[vijaya kumar shankaran (vijayks)]

Hi Anish,

Could you please provide us an update?

Best Regards,
Vijay Kumar

[vivekananda shenoy (vshenoy83)]

Hi Sundar,

Any updates on this issue ? Customer is asking for the updates on this issue.

Are you looking for any other info ?

Regards,
Vivek

[Sundaresan Rajangam (srajanga)]

Hi Vivek,

I haven’t looked at this yet.
I’ll have to replicate the issue locally and debug further.

Thanks,
Sundar

> On Jul 26, 2017, at 4:49 PM, vivekananda shenoy <email address hidden> wrote:
>
> Hi Sundar,
>
> Any updates on this issue ? Customer is asking for the updates on this
> issue.
>
> Are you looking for any other info ?
>
> Regards,
> Vivek
>
> --
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://bugs.launchpad.net/bugs/1701426
>
> Title:
> Incorrect response from analytics api server when queried for number
> of VMIs greater than 48K
>
> Status in Juniper Openstack:
> New
> Status in Juniper Openstack r3.1 series:
> New
> Status in Juniper Openstack r3.2 series:
> New
> Status in Juniper Openstack r4.0 series:
> New
> Status in Juniper Openstack trunk series:
> New
>
> Bug description:
> Customer is running a Curl against analytics api for the number of Virtual Machine interface.
> When the number of VMI interface is less than 48K the response from the Analytics server matches with that of VMI on the setup. However, when there are large number of VMI >48K the response doesn’t match with the existing with VMI and we get different response each time. We get a http 200 OK response in both the case.
> Following response matches with the existing VMI
> Analytics API / configAPI
> 11 k VMI: 11135/11135 OK GET 200 5886 759 12.00 3 60 4
> 30 k VMI: 30000/30000 OK GET 200 15904063 35.785529
> 36 k VMI: 36000/36000 OK GET 200 19084807 45.094492
> 44.3 k VMI: 44327/44327 OK GET 200 23434148 59.721491
> 45 k VMI: 45000/45000 OK GET 200 23826101 62.304583
>
>
> The ones that doesn’t match are
> 46.8 k VMI: 46060/46800 NG GET 200 24316603 59.041701
> 60 k VMI: 55704/60000 NG GET 200 29530343 97.429255
> 177 k VMI: 159975/177061 NG GET 200 84574245 323.764904
>
> Customer is on v3.1.3 Build 75 and the curl command executed is
> curl http://localhost:8081/analytics/uves/virtual-machine-interface/default-domain:*:*?flat
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juniperopenstack/+bug/1701426/+subscriptions

[vivekananda shenoy (vshenoy83)]

Hi Sundar,

Jeba's team is doing a QA on 3.1, I guess you can take help from his team to get hold of a setup and try to reproduce this issue.

Regards,
Vivek

[vivekananda shenoy (vshenoy83)]

Hi Jeba,

Can someone from your team help Sundar to reproduce this issue ?

Regards,
Vivek

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

What is the current status?

Do we have an ETA?

Best Regards,
Vijay Kumar

[Sundaresan Rajangam (srajanga)]

Can you please get the following o/p:

http://<config-ip>:8082/virtual-machine-interfaces
http://<analytics-ip>:8081/analytics/uves/virtual-machine-interfaces
http://<analytics-ip>:8081/analytics/uves/vrouters/*?flat

[vijaya kumar shankaran (vijayks)]

Hi Sundaresan,

Customer responded stating that this setup is no more with them. As its been long time they removed setup thinking that we have all the information to proceed with this bug.

Best Regards,
Vijay Kumar

[Sundaresan Rajangam (srajanga)]

Vijay Kumar,

There is hardly any information in the bug description other than the mismatch between the config and analytics db. Am trying to get help from the Systest team here to recreate the issue to debug further. There is no problem in analytics w.r.t fetching > 48K VMIs

[Anish Mehta (amehta00)]

Closing this bug for now.
We can re-examine this based on scale testing in our lab.

[vijaya kumar shankaran (vijayks)]

Hi Anish,

Customer is following with us for an update. Customer concern is we requesting the information after 2+months and they don't have this setup. They are requesting us to replicate and collect the information. We need to work on this?

I will try to replicate the issue in our lab but not sure if I can hit the same issue.

Best Regards,
Vijay Kumar

[vivekananda shenoy (vshenoy83)]

Hi Sundar,

QA team scale setup is available with the 3.1 code. This setup as 64K VMIs. I tried querying the UVE's and I see that the 2 times I did the queries returned different data. Both times got a 200 OK. But then it is possible that the O/P data has the same UVE's but they are in different order each time we query the UVE's.

However the setup is available and you may have a better way to figure this out. Also can you provide an tentative ETA for this issue as NTTC is constantly asking us the same ?

IP : 10.87.121.77
root/contrail123

Regards,
Vivek

====================

root@5b7s1-1-vm1:~# source /etc/contrail/openstackrc
root@5b7s1-1-vm1:~#
root@5b7s1-1-vm1:~#
root@5b7s1-1-vm1:~# keystone token-get
/usr/lib/python2.7/dist-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/dist-packages/debtcollector/renames.py:45: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py:371: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/session.py:140: DeprecationWarning: keystoneclient.session.Session is deprecated as of the 2.1.0 release in favor of keystoneauth1.session.Session. It will be removed in future releases.
  DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py:56: DeprecationWarning: keystoneclient auth plugins are deprecated as of the 2.1.0 release in favor of keystoneauth1 plugins. They will be removed in future releases.
  'in future releases.', DeprecationWarning)
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2017-08-13T17:52:21Z |
| id | 0610d02e9d964e3f942c96955dd6b88d |
| tenant_id | 715019844b9d4912b15bd8385bf484dd |
| user_id | 8497548cca3845829c2fbe4bf4821fad |
+-----------+----------------------------------+
root@5b7s1-1-vm1:~#
root@5b7s1-1-vm1:~#
root@5b7s1-1-vm1:~# export token="0610d02e9d964e3f942c96955dd6b88d"
root@5b7s1-1-vm1:~#
root@5b7s1-1-vm1:~#
root@5b7s1-1-vm1:~# curl -H "X-Aut...

[Sundaresan Rajangam (srajanga)]

Hi Vivek,

What do you mean by you got different data?
Do you have the o/p from both the queries?

I tried few times. I got the same number of VMI UVEs:

root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2899k 0 0:00:02 0:00:02 --:--:-- 2898k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2774k 0 0:00:02 0:00:02 --:--:-- 2775k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2835k 0 0:00:02 0:00:02 --:--:-- 2835k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 3048k 0 0:00:02 0:00:02 --:--:-- 3049k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2811k 0 0:00:02 0:00:02 --:--:— 2811k
  30911 4616562

Thanks,
Sundar

> On Aug 13, 2017, at 9:59 AM, vivekananda shenoy <email address hidden> wrote:
>
> Hi Sundar,
>
> QA team scale setup is available with the 3.1 code. This setup as 64K
> VMIs. I tried querying the UVE's and I see that the 2 times I did the
> queries returned different data. Both times got a 200 OK. But then it is
> possible that the O/P data has the same UVE's but they are in different
> order each time we query the UVE's.
>
> However the setup is available and you may have a better way to figure
> this out. Also can you provide an tentative ETA for this issue as NTTC
> is constantly asking us the same ?
>
> IP : 10.87.121.77
> root/contrail123
>
> Regards,
> Vivek
>
>
> ====================
>
> root@5b7s1-1-vm1:~# source /etc/contrail/openstackrc
> root@5b7s1-1-vm1:~#
> root@5b7s1-1-vm1:~#
> root@5b7s1-1-vm1:~# keystone token-get
> /usr/lib/python2.7/dist-...

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Could you test this from a different host instead of performing this test from localhost.

Best Regards,
Vijay Kumar

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Sundaresan Rajangam
Sent: Monday, August 14, 2017 11:11 PM
To: Vijay Kumar Shankaran <email address hidden>
Subject: Re: [Bug 1701426] Incorrect response from analytics api server when queried for number of VMIs greater than 48K

Hi Vivek,

What do you mean by you got different data?
Do you have the o/p from both the queries?

I tried few times. I got the same number of VMI UVEs:

root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2899k 0 0:00:02 0:00:02 --:--:-- 2898k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2774k 0 0:00:02 0:00:02 --:--:-- 2775k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2835k 0 0:00:02 0:00:02 --:--:-- 2835k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 3048k 0 0:00:02 0:00:02 --:--:-- 3049k
  30911 4616562
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~#
root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 6541k 100 6541k 0 0 2811k 0 0:00:02 0:00:02 --:--:— 2811k
  30911 4616562

Thanks,
Sundar

> On Aug 13, 2017, at 9:59 AM, vivekananda shenoy <email address hidden> wrote:
>
> Hi Sundar,
>
> QA team scale setup is available with the 3.1 code. This setup as 64K
> VMIs. I tried querying the UVE's and I see that the 2 times I did the
> queries returned different data. Both times got a 200 OK. But then it
> is possible that the O/P data has the same UVE's but they are in
> different order each time we query the...

[Sundaresan Rajangam (srajanga)]

Vijay,

It doesn’t make any difference, whether you query from local or remote host.
Also, vrouter-agent would send the VMI UVEs only if the port is active.
Therefore, if there is a mismatch between the VMIs in config and analytics, then it is possible that some ports may not be active.

If you see this issue again, please collect the following o/p

http://<config-ip>:8082/virtual-machine-interfaces
http://<analytics-ip>:8081/analytics/uves/virtual-machine-interface/*?flat
http://<analytics-ip>:8081/analytics/uves/vrouters/*?flat

The bug can be in "In Progress” state, only if the code is submitted for review.
Am marking this bug as “Incomplete” as we don’t have enough information to proceed further.

Thanks,
Sundar

> On Aug 14, 2017, at 7:04 PM, vijaya kumar shankaran <email address hidden> wrote:
>
> Hi Sundar,
>
> Could you test this from a different host instead of performing this
> test from localhost.
>
> Best Regards,
> Vijay Kumar
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Sundaresan Rajangam
> Sent: Monday, August 14, 2017 11:11 PM
> To: Vijay Kumar Shankaran <email address hidden>
> Subject: Re: [Bug 1701426] Incorrect response from analytics api server when queried for number of VMIs greater than 48K
>
> Hi Vivek,
>
> What do you mean by you got different data?
> Do you have the o/p from both the queries?
>
> I tried few times. I got the same number of VMI UVEs:
>
> root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 6541k 100 6541k 0 0 2899k 0 0:00:02 0:00:02 --:--:-- 2898k
> 30911 4616562
> root@5b7s1-4-vm2:~#
> root@5b7s1-4-vm2:~#
> root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 6541k 100 6541k 0 0 2774k 0 0:00:02 0:00:02 --:--:-- 2775k
> 30911 4616562
> root@5b7s1-4-vm2:~#
> root@5b7s1-4-vm2:~#
> root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 6541k 100 6541k 0 0 2835k 0 0:00:02 0:00:02 --:--:-- 2835k
> 30911 4616562
> root@5b7s1-4-vm2:~#
> root@5b7s1-4-vm2:~#
> root@5b7s1-4-vm2:~# curl -u <> localhost:8181/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep href | wc -lc
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 6541k 100 6541k 0 0 3048k 0 0:00:02 0:0...

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Customer doesn't have this setup anymore. As this more than 2 months old, had this information requested earlier we would have managed to get them as requested.

I will try to replicate this in the lab is there anyother information you are looking for? I will have them collected once the issue is replicable.

Best Regards,
Vijay Kumar

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Could you please share script to automate the creation of VMI?

Best Regards,
Vijay Kumar

[Sundaresan Rajangam (srajanga)]

Hi Vijay,

I don’t have any script. Please check with Jeba.

Thanks,
Sundar

> On Aug 15, 2017, at 7:19 PM, vijaya kumar shankaran <email address hidden> wrote:
>
> Hi Sundar,
>
> Could you please share script to automate the creation of VMI?
>
> Best Regards,
> Vijay Kumar
>
> --
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://bugs.launchpad.net/bugs/1701426
>
> Title:
> Incorrect response from analytics api server when queried for number
> of VMIs greater than 48K
>
> Status in Juniper Openstack:
> Incomplete
> Status in Juniper Openstack r3.1 series:
> Incomplete
> Status in Juniper Openstack r3.2 series:
> Incomplete
> Status in Juniper Openstack r4.0 series:
> Incomplete
> Status in Juniper Openstack trunk series:
> Incomplete
>
> Bug description:
> Customer is running a Curl against analytics api for the number of Virtual Machine interface.
> When the number of VMI interface is less than 48K the response from the Analytics server matches with that of VMI on the setup. However, when there are large number of VMI >48K the response doesn’t match with the existing with VMI and we get different response each time. We get a http 200 OK response in both the case.
> Following response matches with the existing VMI
> Analytics API / configAPI
> 11 k VMI: 11135/11135 OK GET 200 5886 759 12.00 3 60 4
> 30 k VMI: 30000/30000 OK GET 200 15904063 35.785529
> 36 k VMI: 36000/36000 OK GET 200 19084807 45.094492
> 44.3 k VMI: 44327/44327 OK GET 200 23434148 59.721491
> 45 k VMI: 45000/45000 OK GET 200 23826101 62.304583
>
>
> The ones that doesn’t match are
> 46.8 k VMI: 46060/46800 NG GET 200 24316603 59.041701
> 60 k VMI: 55704/60000 NG GET 200 29530343 97.429255
> 177 k VMI: 159975/177061 NG GET 200 84574245 323.764904
>
> Customer is on v3.1.3 Build 75 and the curl command executed is
> curl http://localhost:8081/analytics/uves/virtual-machine-interface/default-domain:*:*?flat
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juniperopenstack/+bug/1701426/+subscriptions

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Customer has provided requested logs for

 http://<config-ip>:8082/virtual-machine-interfaces
 http://<analytics-ip>:8081/analytics/uves/virtual-machine-interfaces
 http://<analytics-ip>:8081/analytics/uves/vrouters/*?flat

They are still noticing this issue.
Currently they are on contrail v3.1.3 Build81.
Config and analytics has been changed from 3 to 9 nodes. Testbed attached.

* http://[analytics node]:8081/analytics/uves/virtual-machine-interface/default-domain:*:*?flat

==============================================
* config-api(virtual-machine-interfaces)
root@JunCom01:~/JN-321_0915# grep uuid openc-41_config.log | wc -l
59077

root@JunCom01:~/JN-321_0915# grep uuid openc-42_config.log | wc -l
59077

root@JunCom01:~/JN-321_0915# grep uuid openc-43_config.log | wc -l
59077

* analytics-api(virtual-machine-interfaces)
root@JunCom01:~/JN-321_0915# grep name openc-51_analytics_vmi.log | wc -l
59077

root@JunCom01:~/JN-321_0915# grep name openc-52_analytics_vmi.log | wc -l
59077

root@JunCom01:~/JN-321_0915# grep name openc-15_analytics_vmi.log | wc -l
59077

* analytics-api detail(virtual-machine-interface/default-domain:*:*?flat)
root@JunCom01:~/JN-321_0915# grep -e "uuid" openc-51_analytics_vmi_all.log | grep -v "vm_uuid" | grep -v "vn_uuid" | wc -l
57667 (-1410)
Difference list : openc41-51_diff.log

root@JunCom01:~/JN-321_0915# grep -e "uuid" openc-52_analytics_vmi_all.log | grep -v "vm_uuid" | grep -v "vn_uuid" | wc -l
57968 (-1109)
Difference list : openc42-52_diff.log

root@JunCom01:~/JN-321_0915# grep -e "uuid" openc-15_analytics_vmi_all.log | grep -v "vm_uuid" | grep -v "vn_uuid" | wc -l
58606 (-471)
Difference list : openc43-15_diff.log

Best Regards,
Vijay Kumar

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

we have the customer setup available. this would be the right time to collect any other information/logs if required. Could you please look into this and update us?

Best Regards,
Vijay Kumar

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Customer has removed the setup for some other requirement.

Could you please share us with the current status?

Best Regards,
Vijay Kumar

[Sundaresan Rajangam (srajanga)]

From the logs #20, I could see query for /analytics/uves/virtual-machine-interface/default-domain:*:*?flat only in openc52. The query was made @ 2017-09-15 12:58:18, 2017-09-15 13:14:25 and 2017-09-15 13:30:51
During this period, I analytics-api was not not able to connect to the redis @ 172.23.10.194

The following Exception was continuously logged in the contrail-analytics-api.log

09/15/2017 01:14:18 PM [contrail-analytics-api]: Exception ConnectionError in uve cache proc. Arguments:
(u'Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.',) : traceback Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 191, in get_cache_uve
    tfilter, ackfilter, False)
  File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 126, in _get_uve_content
    pperes = ppe.execute()
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 2000, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1869, in _execute_transaction
    connection.send_packed_command(all_cmds)
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 292, in send_packed_command
    self.connect()
  File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 235, in connect
    raise ConnectionError(self._error_message(e))
ConnectionError: Error 99 connecting 172.23.10.194:6379. Cannot assign requested address.

So, either there was some connectivity issue to 172.23.10.194 or redis server couldn't accept new connections due to max client connections reached. I had seen the latter case when haproxy continuously connects to redis. Can you please check if the following line is present in haproxy config? If so, please remove this line and restart haproxy and redis service - https://bugs.launchpad.net/juniperopenstack/+bug/1648601

tcp-check connect port 6379

openc52-contrail-analytics-api logs:
-----------------------------------

bash-3.2$ grep -r "virtual-machine-interface" *
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.197 - - [2017-09-15 12:58:18] "GET /analytics/uves/virtual-machine-interface/default-domain:*:*?flat HTTP/1.1" 200 36103651 69.184298
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 13:14:25] "GET /analytics/uves/virtual-machine-interface/default-domain:*:*?flat HTTP/1.1" 200 35523881 72.361021
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.194 - - [2017-09-15 13:30:51] "GET /analytics/uves/virtual-machine-interface/default-domain:*:*?flat HTTP/1.1" 200 35710910 72.576068
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:09] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-011-pr-0949:commonmax-011-vmi-0949001005B-3?flat HTTP/1.1" 200 646 0.002621
log/contrail/contrail-analytics-api-stdout.log.6:172.23.10.193 - - [2017-09-15 16:57:26] "GET /analytics/uves/virtual-machine-interface/default-domain:commonmax-001-pr-0093:commonmax-001-vmi-0093002003B-6?flat HTTP/1.1" 200 642 0.002275
log/contrail/contrail-analytics-api-stdout.log.7:172.23.10.205 - - [2017-09-15 09:31...

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Based on the analysis shared customer re installed the setup and ensured that tcp-check connect port 6379 was not present in haproxy config.

# Get method
----------------------------------------------------------------------
- config API
  /virtual-machine-interfaces

- analytics API
  /analytics/uves/virtual-machine-interface/default-domain:*:*?flat
# Test result (script:4th - 6th)
----------------------------------------------------------------------
-------------------
4th : NG <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
-------------------
masa@ovs-03:~/workspace/ngcl-tools/bin$ ./analytics-stress-per.py -m one
Config API VMI GET
start : 2017-10-25 16:01:37.896985
end : 2017-10-25 16:01:45.279205
vmi count = 59077 <<<<<<<<<<<<<<<<<<<<<<<<<<<Config API
Hit return to continue
Analytics API VMI GET
type = one
vmi count = 36579 <<<<<<<<<<<<<<<<<<<<<<<<<<<Analytics API
start : 2017-10-25 16:01:49.658426
end : 2017-10-25 16:02:33.526010
delta : 0:00:43.867584
max_delay : 0:00:43.867584
min_delay : 0:00:43.867584

-------------------
5th : OK
-------------------
masa@ovs-03:~/workspace/ngcl-tools/bin$ ./analytics-stress-per.py -m one
Config API VMI GET
start : 2017-10-25 16:15:22.235729
end : 2017-10-25 16:15:26.264732
vmi count = 59077
Hit return to continue
Analytics API VMI GET
type = one
vmi count = 59077
start : 2017-10-25 16:15:28.099833
end : 2017-10-25 16:16:39.002510
delta : 0:01:10.902677
max_delay : 0:01:10.902677
min_delay : 0:01:10.902677

The above issue was noticed when the response was from one of the analytics node openc-51
Where the config API returned vmi count = 59077 <<<<<<<<<<<<<<<<<<<<<<<<<<<Config API
And analytics api returned vmi count = 36579 <<<<<<<<<<<<<<<<<<<<<<<<<<<Analytics API
Timestamp
start : 2017-10-25 16:01:49.658426
end : 2017-10-25 16:02:33.526010
The other nodes (openc-52,1 openc-5)returned correct VMI count. At this time
From the openc-51
172.23.10.205 - - [2017-10-25 16:02:27] "GET /analytics/uves/virtual-machine-interface/default-domain:*:*?flat HTTP/1.1" 200 22533655 40.709530
I see discovery client log msg in the openc-51 most of the time repeatedly than the other nodes.

Best Regards,
Vijay Kumar

[vijaya kumar shankaran (vijayks)]

From the node openc-51 contrail-collector.log.1
2017-10-25 Wed 15:56:33:902.624 JST openc-51 [Thread 140630781060864, Pid 31366]: DiscoveryClient [SYS_NOTICE]: DiscoveryClientLogMsg: Message Type: SubscribeResponseHandler Service Name: ApiServer Message: <?xml version="1.0" encoding="utf-8"?>
<response><ApiServer publisher-id="openc-41"><port>9100</port><ip-address>172.23.10.205</ip-address></ApiServer><ApiServer publisher-id="openc-42"><port>9100</port><ip-address>172.23.10.206</ip-address></ApiServer><ApiServer publisher-id="openc-43"><port>9100</port><ip-address>172.23.10.207</ip-address></ApiServer><ttl>578</ttl></response> controller/src/discovery/client/discovery_client.cc 1034

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Do you have an update for us?

Best Regards,
Vijay Kumar

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Customer is pushing us for an update.

Could you please share your analysis?

Best Regards,
Vijay Kumar

[Sundaresan Rajangam (srajanga)]

Please provide access to the setup mentioned in #24

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Customer has provided access to his setup. this is the same setup that you were working with sandeep.

Please let me know when we can setup and work on this?

Best Regards,
Vijay Kumar

[Sundaresan Rajangam (srajanga)]

Hi Vijay,

Can you please send me the testbed details and the procedure to access it?

Thanks,
Sundar

> On Dec 8, 2017, at 4:01 AM, vijaya kumar shankaran <email address hidden> wrote:
>
> Hi Sundar,
>
> Customer has provided access to his setup. this is the same setup that
> you were working with sandeep.
>
> Please let me know when we can setup and work on this?
>
> Best Regards,
> Vijay Kumar
>
> --
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.launchpad.net_bugs_1701426&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=LPHaOrEhcHUkaXTIgszI3jGHWJ2DkgIMvg2FajOezdI&m=0M-iBg7LZG006oEogaIgui-5SjkHY4dRyxmnTwxNOGc&s=cvhNIfxvRR4vJY2oa1Quq_uX8zfnaZqtV-C6yipWPQk&e=
>
> Title:
> Incorrect response from analytics api server when queried for number
> of VMIs greater than 48K
>
> Status in Juniper Openstack:
> Incomplete
> Status in Juniper Openstack r3.1 series:
> Incomplete
> Status in Juniper Openstack r3.2 series:
> Incomplete
> Status in Juniper Openstack r4.0 series:
> Incomplete
> Status in Juniper Openstack r4.1 series:
> Incomplete
> Status in Juniper Openstack trunk series:
> Incomplete
>
> Bug description:
> Customer is running a Curl against analytics api for the number of Virtual Machine interface.
> When the number of VMI interface is less than 48K the response from the Analytics server matches with that of VMI on the setup. However, when there are large number of VMI >48K the response doesn’t match with the existing with VMI and we get different response each time. We get a http 200 OK response in both the case.
> Following response matches with the existing VMI
> Analytics API / configAPI
> 11 k VMI: 11135/11135 OK GET 200 5886 759 12.00 3 60 4
> 30 k VMI: 30000/30000 OK GET 200 15904063 35.785529
> 36 k VMI: 36000/36000 OK GET 200 19084807 45.094492
> 44.3 k VMI: 44327/44327 OK GET 200 23434148 59.721491
> 45 k VMI: 45000/45000 OK GET 200 23826101 62.304583
>
>
> The ones that doesn’t match are
> 46.8 k VMI: 46060/46800 NG GET 200 24316603 59.041701
> 60 k VMI: 55704/60000 NG GET 200 29530343 97.429255
> 177 k VMI: 159975/177061 NG GET 200 84574245 323.764904
>
> Customer is on v3.1.3 Build 75 and the curl command executed is
> curl https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8081_analytics_uves_virtual-2Dmachine-2Dinterface_default-2Ddomain-3A-2A-3A-2A-3Fflat&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=LPHaOrEhcHUkaXTIgszI3jGHWJ2DkgIMvg2FajOezdI&m=0M-iBg7LZG006oEogaIgui-5SjkHY4dRyxmnTwxNOGc&s=mTz0Z6ExYlq_bZM4JKHtQL5Je1zPTVe2tmf5EWEQSdc&e=
>
> To manage notifications about this bug go to:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.launchpad.net_juniperopenstack_-2Bbug_1701426_-2Bsubscriptions&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=LPHaOrEhcHUkaXTIgszI3jGHWJ2DkgIMvg2FajOezdI&m=0M-iBg7LZG006oEogaIgui-5SjkHY4dRyxmnTwxNOGc&s=HNHLyy4VwkF7dWL1q6s1Vf26RR55pzEwuzQRCNNhEYU&e=

[vijaya kumar shankaran (vijayks)]

Hi Sundar,

Mailed you the access details. Please let me know if you need any more information.

Customer tested on 3.1.3.0-85 and are noticing the same issue

The customer's contrail version is 3.1.3.0-85.

* test result
- 1st time
vmi count = 59079 <<<<<<<<<<<<<<<<<<<<<<<<<Config API
vmi count = 53820 <<<<<<<<<<<<<<<<<<<<<<<<<AnalyticsAPI

- 2nd time
vmi count = 59079 <<<<<<<<<<<<<<<<<<<<<<<<<Config API
vmi count = 56590 <<<<<<<<<<<<<<<<<<<<<<<<<AnalyticsAPI

- 3rd time
vmi count = 59079 <<<<<<<<<<<<<<<<<<<<<<<<<Config API
vmi count = 56212 <<<<<<<<<<<<<<<<<<<<<<<<<AnalyticsAPI

Best Regards,
Vijay Kumar

[Sundaresan Rajangam (srajanga)]

The inconsistency in the number of VMI UVEs returned by analytics-api is due to the creation of new redis connection for every UVE request. The system runs out of ephemeral ports and hence not all VMI UVEs are fetched from redis. The patch for https://bugs.launchpad.net/juniperopenstack/+bug/1731187 should fix this issue.