[Build 2572] many cases involving svc-mon failed during sanity

Bug #1443055 reported by Vedamurthy Joshi
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Committed
High
Rudra Rugge
R2.20
Fix Committed
High
Rudra Rugge

Bug Description

Build 3.0 2572 all distros/skus

Many exceptions with below bt were seen across multiple sanity jobs. Most cases involving svc-mon seem to have failed due to this.

For ex: On nodei31, single-node sanity setup:

root@nodei31:~/contrail-test/logs# grep "END TEST" * | grep FAIL
testecmpsanity.log:2015-04-12 12:44:25,060 - INFO - END TEST : test_ecmp_svc_in_network_with_3_instance : FAILED[0:11:22]
testecmpsanity.log:2015-04-12 12:51:48,121 - INFO - END TEST : test_ecmp_svc_in_network_with_static_route_no_policy : FAILED[0:07:23]
testecmpsanity.log:2015-04-12 12:59:25,053 - INFO - END TEST : test_ecmp_svc_transparent_with_3_instance : FAILED[0:07:37]
testecmpwithsvmchange.log:2015-04-12 13:39:12,258 - INFO - END TEST : test_ecmp_with_svm_deletion : FAILED[1:04:37]
testheat.log:2015-04-12 13:11:44,538 - INFO - END TEST : test_transit_vn_with_svc : FAILED[0:08:39]
testmultiinlinesvc.log:2015-04-12 13:47:23,697 - INFO - END TEST : test_three_stage_SC : FAILED[0:07:35]
testpolicyacl.log:2015-04-12 14:32:26,806 - INFO - END TEST : test_policy_cidr_src_vn_dst_cidr : FAILED[1:40:21]
testpolicyacl.log:2015-04-12 16:12:47,931 - INFO - END TEST : test_policy_inheritance_src_pol_dst_any : FAILED[1:40:21]
testports.log:2015-04-12 12:45:00,687 - INFO - END TEST : test_aap_with_vrrp_priority_change : FAILED[0:05:28]
testports.log:2015-04-12 12:46:28,445 - INFO - END TEST : test_ports_attach_detach : FAILED[0:01:28]
testroutersnat.log:2015-04-12 13:16:47,512 - INFO - END TEST : test_basic_snat_behavior : FAILED[0:04:25]
testsvcmirror.log:2015-04-12 13:54:28,173 - INFO - END TEST : test_svc_mirroring : FAILED[0:05:07]
testsvcregr.log:2015-04-12 13:27:23,639 - INFO - END TEST : test_svc_in_network_datapath : FAILED[0:06:15]
testsvcregr.log:2015-04-12 13:32:40,117 - INFO - END TEST : test_svc_in_network_nat_private_to_public : FAILED[0:05:17]
testsvcregr.log:2015-04-12 13:40:16,546 - INFO - END TEST : test_svc_monitor_datapath : FAILED[0:07:36]
testsvcregr.log:2015-04-12 13:48:49,851 - INFO - END TEST : test_svc_transparent_with_3_instance : FAILED[0:08:33]
root@nodei31:~/contrail-test/logs#

Logs will be in http://10.204.216.50/Docs/bugs/#

<type 'exceptions.RuntimeError'>
Python 2.7.3: /usr/bin/python
Sun Apr 12 13:54:13 2015

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py in launch_timer(monitor=<svc_monitor.svc_monitor.SvcMonitor object>)
  787 gevent.sleep(svc_info.get_vm_health_interval())
  788 try:
  789 timer_callback(monitor)
  790 except Exception:
  791 cgitb_error_log(monitor)
global timer_callback = <function timer_callback>
monitor = <svc_monitor.svc_monitor.SvcMonitor object>

 /usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py in timer_callback(monitor=<svc_monitor.svc_monitor.SvcMonitor object>)
  774 continue
  775
  776 for vn_id in project.virtual_networks:
  777 vn = VirtualNetworkSM.get(vn_id)
  778 if not vn or vn.virtual_machine_interfaces:
project = <svc_monitor.config_db.ProjectSM object>
project.virtual_networks = set([u'f182d000-69e2-4ba3-9f85-c332113462b2', u'f20b6aff-3dee-4833-ac13-507bddfb24b7'])
<type 'exceptions.RuntimeError'>: Set changed size during iteration
    __class__ = <type 'exceptions.RuntimeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.RuntimeError object>
    __dict__ = {}
    __doc__ = 'Unspecified run-time error.'
    __format__ = <built-in method __format__ of exceptions.RuntimeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.RuntimeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.RuntimeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.RuntimeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.RuntimeError object>
    __init__ = <method-wrapper '__init__' of exceptions.RuntimeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.RuntimeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.RuntimeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.RuntimeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.RuntimeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.RuntimeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.RuntimeError object>
    __str__ = <method-wrapper '__str__' of exceptions.RuntimeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.RuntimeError object>
    args = ('Set changed size during iteration',)
    message = 'Set changed size during iteration'

The above is a description of an error in a Python program. Here is
the original traceback:

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 789, in launch_timer
    timer_callback(monitor)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 776, in timer_callback
    for vn_id in project.virtual_networks:
RuntimeError: Set changed size during iteration

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : master

Review in progress for https://review.opencontrail.org/9062
Submitter: Rudra Rugge (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/9062
Committed: http://github.org/Juniper/contrail-controller/commit/a74d376f66fe8cf5e9a0050d69f887700240f0f3
Submitter: Zuul
Branch: master

commit a74d376f66fe8cf5e9a0050d69f887700240f0f3
Author: Rudra Rugge <email address hidden>
Date: Sun Apr 12 23:02:24 2015 -0700

Copy shared vn list during deletion

Copy shared VN list from a set into a list so that
the shared VNs can be deleted without causing a delete
excpetion on the iteration of the set.

Change-Id: I911da59efcfb3004839e1259a0e2fb1b8184d67c
Closes-Bug: #1443055

Changed in juniperopenstack:
status: New → Fix Committed
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : R2.20

Review in progress for https://review.opencontrail.org/9358
Submitter: Rudra Rugge (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/9358
Committed: http://github.org/Juniper/contrail-controller/commit/db907d614fe04b05aa020a22e622eb10d285f64e
Submitter: Zuul
Branch: R2.20

commit db907d614fe04b05aa020a22e622eb10d285f64e
Author: Rudra Rugge <email address hidden>
Date: Sun Apr 12 23:02:24 2015 -0700

Copy shared vn list during deletion

Copy shared VN list from a set into a list so that
the shared VNs can be deleted without causing a delete
excpetion on the iteration of the set.

Change-Id: I2674e057443065a8505e8a4eb5d45efed942b4bb
Closes-Bug: #1443055

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.