Juniper Openstack

schema transformer could crash during port update if port disappears in the middle of the update

Bug #1691590 reported by Sachin Bansal on 2017-05-17

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.21.x	Fix Committed	Undecided	Sahil Sabharwal	Juniper Openstack r2.21.3
R4.0	Fix Committed	Undecided	Sahil Sabharwal	Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk	Fix Committed	Undecided	Sahil Sabharwal	Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

Observed on a customer setup that schema transformer could crash with the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 3870, in launch_arc
    transformer.process_poll_result(result)
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 3445, in process_poll_result
    func(idents, meta)
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 3113, in add_virtual_machine_interface_virtual_network
    vmi.set_virtual_network(vn_name)
  File "/usr/lib/python2.6/site-packages/schema_transformer/to_bgp.py", line 2323, in set_virtual_network
    _vnc_lib.virtual_machine_interface_update(self.obj)
  File "/usr/lib/python2.6/site-packages/vnc_api/gen/vnc_api_client_gen.py", line 4340, in virtual_machine_interface_update
    self.ref_update('virtual-machine-interface', obj.uuid, ref_name, ref[0], list(ref[1]), 'ADD', ref[2])
  File "/usr/lib/python2.6/site-packages/vnc_api/vnc_api.py", line 577, in ref_update
    raise he
HttpError: HTTP Status: 500 Content: Internal Server Error

This happened because api server didn't handle the disappearance of the port while it tried to update last modified timestamp with the traceback below:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bottle.py", line 763, in _handle
    return route.call(**args)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1624, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/bottle.py", line 1574, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 496, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 659, in ref_update_http_post
    id = self._db_conn.ref_update(obj_type, obj_uuid, ref_type, ref_uuid, {'attr': attr}, operation)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1540, in ref_update
    ref_data, operation)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 550, in ref_update
    self.update_last_modified(bch, obj_uuid)
  File "/usr/lib/python2.6/site-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 675, in update_last_modified
    ['prop:id_perms'])['prop:id_perms'])
  File "/usr/lib/python2.6/site-packages/cfgm_common/vnc_cassandra.py", line 108, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pycassa/columnfamily.py", line 652, in get
    read_consistency_level or self.read_consistency_level)
  File "/usr/lib/python2.6/site-packages/pycassa/pool.py", line 577, in execute
    return getattr(conn, f)(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pycassa/pool.py", line 127, in new_f
    result = f(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pycassa/cassandra/Cassandra.py", line 534, in get
    return self.recv_get()
  File "/usr/lib/python2.6/site-packages/pycassa/cassandra/Cassandra.py", line 561, in recv_get
    raise result.nfe
NotFoundException: NotFoundException(_message=None)
10.96.0.155 - - [2017-05-03 15:16:47] "POST /ref-update HTTP/1.1" 500 156 0.019676

Tags:

Revision history for this message

Sachin Bansal (sbansal) wrote on 2017-05-17:

On releases older than 3.0, such a crash could cause schema transformer to lose some 'in-flight' messages from ifmap server. As a result, those messages won't be processed resulting in lost functionality until schema transformer is restarted.

This problem is not there in 3.0 or later because schema transformer does not drop any in-flight messages. However, the traceback still needs to be handled in api server.

information type:

Proprietary → Public

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-06-07: [Review update] master

Review in progress for https://review.opencontrail.org/32643
Submitter: <email address hidden> (<email address hidden>)

Vineet Gupta (vineetrf) on 2017-06-09

tags:

added: wpc

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-06-09: A change has been merged

Reviewed: https://review.opencontrail.org/32643
Committed: http://github.com/Juniper/contrail-controller/commit/42eb794b6ce51b6d9034e9214b7f097451203709
Submitter: Zuul (<email address hidden>)
Branch: master

commit 42eb794b6ce51b6d9034e9214b7f097451203709
Author: Sahil <email address hidden>
Date: Wed Jun 7 15:47:53 2017 -0700

[api_server]: Fix to call ref_update function.

1. Added a call to ref_update function in _put_common().
We were only doing dbe_update at the time of ref_update_http_post.

2. Modified update_last_modified function for cassandra so that it
doesn't throw a traceback when obj. gets deleted at the time
of obj. update

Change-Id: I36064a8b07503732ed76a4ffaddef9ef02a3c054
Closes-Bug: 1691590

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-06-09: [Review update] R4.0

Review in progress for https://review.opencontrail.org/32719
Submitter: <email address hidden> (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-06-09: [Review update] R2.21.x

#11

Review in progress for https://review.opencontrail.org/32726
Submitter: <email address hidden> (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-06-13: A change has been merged

#13

Reviewed: https://review.opencontrail.org/32719
Committed: http://github.com/Juniper/contrail-controller/commit/38e3d3e30925ca465dffbdef1bd4b5f9d8f83bb5
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 38e3d3e30925ca465dffbdef1bd4b5f9d8f83bb5
Author: Sahil <email address hidden>
Date: Wed Jun 7 15:47:53 2017 -0700

[api_server]: Fix to call ref_update function.

1. Added a call to ref_update function in _put_common().
We were only doing dbe_update at the time of ref_update_http_post.

2. Modified update_last_modified function for cassandra so that it
doesn't throw a traceback when obj. gets deleted at the time
of obj. update

Change-Id: I36064a8b07503732ed76a4ffaddef9ef02a3c054
Closes-Bug: 1691590
(cherry picked from commit 42eb794b6ce51b6d9034e9214b7f097451203709)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-06-14:

#14

Reviewed: https://review.opencontrail.org/32726
Committed: http://github.com/Juniper/contrail-controller/commit/2a8ebf97e7f606563fb866b5cc777d50f4d83cb2
Submitter: Zuul (<email address hidden>)
Branch: R2.21.x

commit 2a8ebf97e7f606563fb866b5cc777d50f4d83cb2
Author: Sahil <email address hidden>
Date: Fri Jun 9 14:44:33 2017 -0700

[api_server]: Fix to call ref_update function.

1. Added a call to ref_update function in _put_common().
We were only doing dbe_update at the time of ref_update_http_post.

2. Modified update_last_modified function for cassandra so that it
doesn't throw a traceback when obj. gets deleted at the time
of obj. update

Change-Id: I2b58b4a4974891a5b584ad7848f8747fe5a8f8d8
Closes-Bug: 1691590

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.