Lbaas Healthmonitor not working
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R3.2 |
Fix Committed
|
High
|
Yuvaraja Mariappan | |||
R4.0 |
Fix Committed
|
High
|
Yuvaraja Mariappan | |||
R4.1 |
Fix Committed
|
High
|
Yuvaraja Mariappan | |||
Trunk |
Fix Committed
|
High
|
Yuvaraja Mariappan |
Bug Description
Begin forwarded message:
From: Nimit Pathak <email address hidden>
Date: November 15, 2017 at 8:42:53 AM PST
To: Ignatious Johnson <email address hidden>, Senthilnathan Murugappan <email address hidden>, Ping Song <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>, Sachchidanand Vaidya <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Hi Ignatious/Yuvaraja,
I dug thru all the config files related to keystone and loadbalancer, but didn’t find anything odd… but I also observed the following error in neutron server.log, maybe this is an indication of something missing ? Can you see if you can find anything if I miss? attached is the curl o/p for you reference and also neutron and server logs. Please let me know if you need any additional information.
Regards,
-Nimit
$ root@mtn6nova01:~# neutron --d lb-healthmonito
DEBUG: keystoneclient.
DEBUG: keystoneclient.
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: neutronclient.
DEBUG: keystoneclient.
DEBUG: keystoneclient.
DEBUG: keystoneclient.
DEBUG: neutronclient.
ERROR: neutronclient.shell Request Failed: internal server error while processing your request.
Traceback (most recent call last):
File "/usr/lib/
return run_command(cmd, cmd_parser, sub_argv)
File "/usr/lib/
return cmd.run(known_args)
File "/usr/lib/
return super(OpenStack
File "/usr/lib/
column_names, data = self.take_
File "/usr/lib/
return self.get_
File "/usr/lib/
data = self.retrieve_
File "/usr/lib/
data = self.call_
File "/usr/lib/
data = obj_lister(
File "/usr/lib/
ret = self.function(
File "/usr/lib/
retrieve_all, **_params)
File "/usr/lib/
for r in self._paginatio
File "/usr/lib/
res = self.get(path, params=params)
File "/usr/lib/
headers=
File "/usr/lib/
headers=
File "/usr/lib/
self.
File "/usr/lib/
exception_
File "/usr/lib/
status_
InternalServerE
/var/log/
2017-11-15 06:19:43.449 23102 DEBUG keystoneclient.
2017-11-15 06:19:44.309 23102 DEBUG keystoneclient.
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "2b368c36d00e46
_http_log_response /usr/lib/
2017-11-15 06:19:45.151 23096 INFO neutron.wsgi [-] (23096) accepted ('135.25.12.158', 52187)
2017-11-15 06:19:45.211 23102 ERROR neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.215 23102 INFO neutron.wsgi [req-7d8a17d7-
/var/log/
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
return self._send_
File "/usr/lib/
return send_method(data, *args)
error: [Errno 90] Message too long
Logged from file contrail_plugin.py, line 383
/var/log/
WARNING:
172.29.26.41 - - [2017-11-15 05:48:08] "GET /global-
172.29.26.107 - - [2017-11-15 05:48:09] "POST /neutron/port HTTP/1.1" 200 866987 6.547260
172.29.26.107 - - [2017-11-15 05:48:09] "GET / HTTP/1.1" 200 18293 0.031766
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:13] "GET / HTTP/1.1" 200 18293 0.031076
172.29.26.41 - - [2017-11-15 05:48:14] "GET / HTTP/1.1" 200 18293 0.034103
WARNING:
172.29.26.41 - - [2017-11-15 05:48:14] "GET /physical-
172.29.26.41 - - [2017-11-15 05:48:15] "GET /physical-
172.29.26.41 - - [2017-11-15 05:48:15] "GET /physical-
$ root@mtn6nova01:~# neutron --d lb-healthmonito
DEBUG: keystoneclient.
DEBUG: keystoneclient.
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: neutronclient.
DEBUG: keystoneclient.
DEBUG: keystoneclient.
DEBUG: keystoneclient.
DEBUG: neutronclient.
ERROR: neutronclient.shell Request Failed: internal server error while processing your request.
Traceback (most recent call last):
File "/usr/lib/
return run_command(cmd, cmd_parser, sub_argv)
File "/usr/lib/
return cmd.run(known_args)
File "/usr/lib/
return super(OpenStack
File "/usr/lib/
column_names, data = self.take_
File "/usr/lib/
return self.get_
File "/usr/lib/
data = self.retrieve_
File "/usr/lib/
data = self.call_
File "/usr/lib/
data = obj_lister(
File "/usr/lib/
ret = self.function(
File "/usr/lib/
retrieve_all, **_params)
File "/usr/lib/
for r in self._paginatio
File "/usr/lib/
res = self.get(path, params=params)
File "/usr/lib/
headers=
File "/usr/lib/
headers=
File "/usr/lib/
self.
File "/usr/lib/
exception_
File "/usr/lib/
status_
InternalServerE
/var/log/
2017-11-15 06:19:43.449 23102 DEBUG keystoneclient.
2017-11-15 06:19:44.309 23102 DEBUG keystoneclient.
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "2b368c36d00e46
_http_log_response /usr/lib/
2017-11-15 06:19:45.151 23096 INFO neutron.wsgi [-] (23096) accepted ('135.25.12.158', 52187)
2017-11-15 06:19:45.211 23102 ERROR neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.211 23102 TRACE neutron.
2017-11-15 06:19:45.215 23102 INFO neutron.wsgi [req-7d8a17d7-
/var/log/
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
return self._send_
File "/usr/lib/
return send_method(data, *args)
error: [Errno 90] Message too long
Logged from file contrail_plugin.py, line 383
/var/log/
WARNING:
172.29.26.41 - - [2017-11-15 05:48:08] "GET /global-
172.29.26.107 - - [2017-11-15 05:48:09] "POST /neutron/port HTTP/1.1" 200 866987 6.547260
172.29.26.107 - - [2017-11-15 05:48:09] "GET / HTTP/1.1" 200 18293 0.031766
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:11] "GET /loadbalancer-
172.29.26.107 - - [2017-11-15 05:48:13] "GET / HTTP/1.1" 200 18293 0.031076
172.29.26.41 - - [2017-11-15 05:48:14] "GET / HTTP/1.1" 200 18293 0.034103
WARNING:
172.29.26.41 - - [2017-11-15 05:48:14] "GET /physical-
172.29.26.41 - - [2017-11-15 05:48:15] "GET /physical-
172.29.26.41 - - [2017-11-15 05:48:15] "GET /physical-
From: Nimit Pathak <email address hidden>
Date: Tuesday, November 14, 2017 at 10:45 AM
To: Ignatious Johnson <email address hidden>, Senthilnathan Murugappan <email address hidden>, Ping Song <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>, Sachchidanand Vaidya <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Hi All,
Any update on Ignatious comments. Please let me know.
Regards,
-Nimit
From: Ignatious Johnson <email address hidden>
Date: Wednesday, November 8, 2017 at 7:05 PM
To: Nimit Pathak <email address hidden>, Senthilnathan Murugappan <email address hidden>, Ping Song <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>, Sachchidanand Vaidya <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
+ Sachin Vaidya
Hi Yuvaraj,
For one of the health monitor I don’t see loadbalancer_
Is it expected to create a health monitor without it?
If Yes, we are not handling it in the code https:/
If No, explain here why this attribute is mandatory?
Attached the o/p of loadbalancer-
{
],
},
},
}
},
Thanks,
Ignatious
From: Nimit Pathak <email address hidden>
Date: Tuesday, November 7, 2017 at 12:59 PM
To: Ignatious Johnson <email address hidden>, Senthilnathan Murugappan <email address hidden>, Ping Song <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Hi Ignatious,
Yes I had tried V1 as suggested by Yuvaraj.
configured to use v1 lb api in our env.
root@mtn6nova01:~# neutron ext-list
+------
| alias | name |
+------
| contrail | Contrail Extension |
| security-group | security-group |
| ipam | Network IP Address Management |
| port-security | Port Security |
| binding | Port Binding |
| provider | Provider Network |
| agent | agent |
| quotas | Quota management support |
| route-table | route-table |
| extra_lbaas_opts | Loadbalancer as a Service |
| external-net | Neutron external network |
| policy | Network Policy |
| router | Neutron L3 Router |
| allowed-
| extra_dhcp_opt | Neutron Extra DHCP opts |
| lbaas | LoadBalancing service |
Thanks,
-Nimit
From: Ignatious Johnson <email address hidden>
Date: Monday, November 6, 2017 at 4:32 PM
To: Senthilnathan Murugappan <email address hidden>, Ping Song <email address hidden>, Nimit Pathak <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Hi Nimit,
Did we try the steps provided by Yuvaraj?
Can you give access to the setup
or
Can you share the neutron-server log and contrail-api log during "neutron --d lb-healthmonito
Thanks,
Ignatious
From: Senthilnathan Murugappan <email address hidden>
Date: Monday, November 6, 2017 at 1:49 PM
To: Ping Song <email address hidden>, Nimit Pathak <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>, Ignatious Johnson <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Hi Ignatious,
Would you be able to check it out?
Thanks,
Senthil
From: Ping Song <email address hidden>
Date: Monday, November 6, 2017 at 1:24 PM
To: Nimit Pathak <email address hidden>, Senthilnathan Murugappan <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: RE: Does Contrail 3.2.x support lb-healthmonitor-* api?
Senthil:
The the workaround does NOT work at all in att setup.
And they are with 3.2.5.
They have used this command:
AUTHN_TOKEN_URL = /v2.0/tokens
to replace:
AUTHN_URL = /v2.0/tokens
And restarted these services:
contrail-
contrail-api
they still see the exact issue.
Any idea?
Regards
ping
From: Nimit Pathak
Sent: Thursday, October 26, 2017 4:26 PM
To: Senthilnathan Murugappan <email address hidden>; Yuvaraja Mariappan <email address hidden>; Sachin Bansal <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Thanks Senthil. I will try it in my lab.
Regards,
-Nimit
From: Senthilnathan Murugappan <email address hidden>
Date: Thursday, October 26, 2017 at 1:04 PM
To: Yuvaraja Mariappan <email address hidden>, Sachin Bansal <email address hidden>, Nimit Pathak <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Nimit,
If using R3.2.6 its due to https:/
The bug has the WorkAround specified in comment#1
- Senthil
From: Yuvaraja Mariappan <email address hidden>
Date: Wednesday, October 25, 2017 at 2:44 PM
To: Sachin Bansal <email address hidden>, Nimit Pathak <email address hidden>
Cc: ask-contrail <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
Hi nimit,
All lb-* commands are version 1.
To use v1 commands, change the driver from v2 to v1 in neutron.conf and restart the neutron-server.
If you any issues, let me know.
Thanks,
Yuvaraja
From: Sachin Bansal <email address hidden>
Date: Thursday, October 26, 2017 at 2:19 AM
To: Nimit Pathak <email address hidden>
Cc: ask-contrail <email address hidden>, Yuvaraja Mariappan <email address hidden>
Subject: Re: Does Contrail 3.2.x support lb-healthmonitor-* api?
[+Yuvaraja]
On Oct 25, 2017, at 1:42 PM, Nimit Pathak <email address hidden> wrote:
Hi All,
Any update on this issue.
Please let me know.
Regards,
-Nimit
From: Nimit Pathak <email address hidden>
Date: Tuesday, October 24, 2017 at 1:24 PM
To: ask-contrail <email address hidden>
Subject: Does Contrail 3.2.x support lb-healthmonitor-* api?
Hi Experts,
I try that in lab and I am getting a internal server error…. All the lb-pool-* api work, but not the lb-healthmonitor . I have tried healthmontoir, and the first step is to go to CLI on MOS controller, and try to list existing ones, if any… here is a traceback with debug. So is the lb-healthmonitor functionality supported?
~$ neutron --d lb-healthmonito
DEBUG: keystoneclient.
DEBUG: keystoneclient.
RESP BODY: {"version": {"status": "stable", "updated": "2015-03-
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: stevedore.extension found extension EntryPoint.
DEBUG: neutronclient.
DEBUG: keystoneclient.
DEBUG: keystoneclient.
DEBUG: keystoneclient.
DEBUG: neutronclient.
ERROR: neutronclient.shell Request Failed: internal server error while processing your request.
Traceback (most recent call last):
File "/usr/lib/
return run_command(cmd, cmd_parser, sub_argv)
File "/usr/lib/
return cmd.run(known_args)
File "/usr/lib/
return super(OpenStack
File "/usr/lib/
column_names, data = self.take_
File "/usr/lib/
return self.get_
File "/usr/lib/
data = self.retrieve_
File "/usr/lib/
data = self.call_
File "/usr/lib/
data = obj_lister(
File "/usr/lib/
ret = self.function(
File "/usr/lib/
retrieve_all, **_params)
File "/usr/lib/
for r in self._paginatio
File "/usr/lib/
res = self.get(path, params=params)
File "/usr/lib/
headers=
File "/usr/lib/
headers=
File "/usr/lib/
self.
File "/usr/lib/
exception_
File "/usr/lib/
status_
InternalServerE
Please let me know.
Regards,
-Nimit
information type: | Proprietary → Private |
tags: | added: lbaas |
Changed in juniperopenstack: | |
importance: | Undecided → High |
assignee: | nobody → Yuvaraja Mariappan (ymariappan) |
information type: | Private → Proprietary |
information type: | Proprietary → Private |
information type: | Private → Public |
Review in progress for https:/ /review. opencontrail. org/38400
Submitter: Yuvaraja Mariappan