| 2017-02-16 23:25:37 |
Piyush Srivastava |
bug |
|
|
added bug |
| 2017-02-16 23:25:52 |
Piyush Srivastava |
bug |
|
|
added subscriber wpc-contrail |
| 2017-02-21 21:00:25 |
Piyush Srivastava |
description |
After reboot of contrail services, it looks like contrail-schema tries to clean up stale routing instance objects. We were seeing a lot of errors related to schema not able to delete routing instance objects.
02/10/2017 08:54:41 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttint.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
02/10/2017 08:54:43 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttprod.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
02/10/2017 08:59:34 PM [contrail-schema]: Starting Introspect on HTTP Port 8087
02/10/2017 08:59:34 PM [contrail-schema]: Cannot write http_port 8087 to /tmp/contrail-schema.2826.http_port
02/10/2017 08:59:39 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttint.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
02/10/2017 08:59:39 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttprod.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
On closer inspection, we found out that the routing instance objects had 'fq_name' attribute missing which caused schema to throw exceptions and crash. As a side effect of this, tap interfaces for new VMs on openstack were not receiving a vrf and showing in ERROR state. To work around this problem we added the following patch to
/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py
1313 obj_dict = self._db_conn.uuid_to_obj_dict(uuid)
1314 if 'fq_name' not in obj_dict: # patched line
1315 return (True, '') # patched line
1316 parent_fq_name = json.loads(obj_dict['fq_name'])[:-1]
1317 try:
1318 parent_uuid = self._db_conn.fq_name_to_uuid(
1319 parent_type, parent_fq_name)
1320 except NoIdError:
Why are the routing instance getting into a corrputed state and what is the proper fix for this issue? |
Contrail version: 2.21.3-56
After reboot of contrail services, it looks like contrail-schema tries to clean up stale routing instance objects. We were seeing a lot of errors related to schema not able to delete routing instance objects. The impact
of this issue is that new virtual machine interfaces are not receiving
a routing instance object.
02/10/2017 08:54:41 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttint.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
02/10/2017 08:54:43 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttprod.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
02/10/2017 08:59:34 PM [contrail-schema]: Starting Introspect on HTTP Port 8087
02/10/2017 08:59:34 PM [contrail-schema]: Cannot write http_port 8087 to /tmp/contrail-schema.2826.http_port
02/10/2017 08:59:39 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttint.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
02/10/2017 08:59:39 PM [contrail-schema]: Error while deleting routing instance default-domain:wd5-ttprod.az2.eng.pdx.wd:e2:e2: HTTP Status: 500 Content: Internal Server Error
On closer inspection, we found out that the routing instance objects had 'fq_name' attribute missing which caused schema to throw exceptions and crash. As a side effect of this, tap interfaces for new VMs on openstack were not receiving a vrf and showing in ERROR state. To work around this problem we added the following patch to
/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py
1313 obj_dict = self._db_conn.uuid_to_obj_dict(uuid)
1314 if 'fq_name' not in obj_dict: # patched line
1315 return (True, '') # patched line
1316 parent_fq_name = json.loads(obj_dict['fq_name'])[:-1]
1317 try:
1318 parent_uuid = self._db_conn.fq_name_to_uuid(
1319 parent_type, parent_fq_name)
1320 except NoIdError:
Why are the routing instance getting into a corrputed state and what is the proper fix for this issue? |
|
| 2017-02-23 21:01:28 |
Ashish Ranjan |
bug task added |
|
juniperopenstack |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
nominated for series |
|
juniperopenstack/r2.21.x |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
bug task added |
|
juniperopenstack/r2.21.x |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
nominated for series |
|
juniperopenstack/r3.2 |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
bug task added |
|
juniperopenstack/r3.2 |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
nominated for series |
|
juniperopenstack/trunk |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
bug task added |
|
juniperopenstack/trunk |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
nominated for series |
|
juniperopenstack/r3.0 |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
bug task added |
|
juniperopenstack/r3.0 |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
nominated for series |
|
juniperopenstack/r3.1 |
|
| 2017-02-23 21:01:51 |
Ashish Ranjan |
bug task added |
|
juniperopenstack/r3.1 |
|
| 2017-02-23 21:02:06 |
Ashish Ranjan |
juniperopenstack/r2.21.x: assignee |
|
Rudra Rugge (rudrarugge) |
|
| 2017-03-06 22:46:58 |
Sachin Bansal |
juniperopenstack/r3.0: status |
New |
Fix Committed |
|
| 2017-03-06 22:47:01 |
Sachin Bansal |
juniperopenstack/r3.1: status |
New |
Fix Committed |
|
| 2017-03-06 22:47:06 |
Sachin Bansal |
juniperopenstack/r3.2: status |
New |
Fix Committed |
|
| 2017-03-06 22:47:09 |
Sachin Bansal |
juniperopenstack/trunk: status |
New |
Fix Committed |
|
| 2017-03-06 22:47:12 |
Sachin Bansal |
opencontrail: status |
New |
Fix Committed |
|
| 2017-03-06 23:33:14 |
Sachin Bansal |
juniperopenstack/r2.21.x: assignee |
Rudra Rugge (rudrarugge) |
Sachin Bansal (sbansal) |
|
| 2017-03-06 23:45:40 |
OpenContrail Admin |
juniperopenstack/r2.21.x: status |
New |
In Progress |
|
| 2017-03-07 04:26:35 |
OpenContrail Admin |
juniperopenstack/r2.21.x: status |
In Progress |
Fix Committed |
|
| 2017-03-07 04:26:37 |
OpenContrail Admin |
juniperopenstack/r2.21.x: milestone |
|
r2.21.3 |
|
| 2017-03-23 17:54:57 |
Jeba Paulaiyan |
tags |
|
config |
|
