Session's persistent objects across transactions create inconsistency

Bug #1404412 reported by Sumit Naiksatam
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Group Based Policy
Fix Released
High
Ivar Lazzaro

Bug Description

In GBP, we happen to run many consecutive transactions using the same session object (when calling Neutron's plugins for creating implicit objects for example).

This can sometimes create inconsistencies between what is persisted in the session and the actual DB state. using expunge_all() before such operations will make sure that the session object will refresh its state.

An example is what sporadically happens with the following UT:

gbp.neutron.tests.unit.services.grouppolicy.test_resource_mapping.TestPolicyRuleSet.test_update_policy_classifier_hier

gbp.neutron.tests.unit.services.grouppolicy.test_resource_mapping.TestPolicyRuleSet.test_update_policy_classifier_hier_StringException: Traceback (most recent call last):
_StringException: Empty attachments:
  pythonlogging:'neutron.api.extensions'
  stderr
  stdout

pythonlogging:'': {{{
2014-12-19 20:04:23,280 INFO [neutron.manager] Loading core plugin: gbp.neutron.tests.unit.services.grouppolicy.test_resource_mapping.NoL3NatSGTestPlugin
2014-12-19 20:04:23,281 INFO [neutron.manager] Loading Plugin: gbp.neutron.services.grouppolicy.plugin.GroupPolicyPlugin
2014-12-19 20:04:23,281 INFO [gbp.neutron.services.grouppolicy.extension_manager] Configured extension driver names: []
2014-12-19 20:04:23,282 INFO [gbp.neutron.services.grouppolicy.extension_manager] Loaded extension driver names: []
2014-12-19 20:04:23,282 INFO [gbp.neutron.services.grouppolicy.extension_manager] Registered extension drivers: []
2014-12-19 20:04:23,282 INFO [gbp.neutron.services.grouppolicy.policy_driver_manager] Configured policy driver names: ['implicit_policy', 'resource_mapping']
2014-12-19 20:04:23,283 INFO [gbp.neutron.services.grouppolicy.policy_driver_manager] Loaded policy driver names: ['implicit_policy', 'resource_mapping']
2014-12-19 20:04:23,283 INFO [gbp.neutron.services.grouppolicy.policy_driver_manager] Registered policy drivers: ['implicit_policy', 'resource_mapping']
2014-12-19 20:04:23,283 INFO [gbp.neutron.services.grouppolicy.policy_driver_manager] Initializing policy driver 'implicit_policy'
2014-12-19 20:04:23,284 INFO [gbp.neutron.services.grouppolicy.policy_driver_manager] Initializing policy driver 'resource_mapping'
2014-12-19 20:04:23,284 INFO [neutron.manager] Loading Plugin: router
2014-12-19 20:04:23,285 INFO [neutron.db.l3_agentschedulers_db] Skipping period L3 agent status check because automatic router rescheduling is disabled.
2014-12-19 20:04:23,285 INFO [neutron.manager] Loading Plugin: gbp.neutron.services.servicechain.servicechain_plugin.ServiceChainPlugin
2014-12-19 20:04:23,286 INFO [gbp.neutron.services.servicechain.driver_manager] Configured servicechain driver names: ['dummy']
2014-12-19 20:04:23,286 INFO [gbp.neutron.services.servicechain.driver_manager] Loaded servicechain driver names: ['dummy']
2014-12-19 20:04:23,286 INFO [gbp.neutron.services.servicechain.driver_manager] Registered servicechain drivers: ['dummy']
2014-12-19 20:04:23,286 INFO [gbp.neutron.services.servicechain.driver_manager] Initializing servicechain driver 'dummy'
2014-12-19 20:04:23,310 INFO [neutron.common.config] Config paste file: /home/jenkins/workspace/gate-group-based-policy-python27/.tox/py27/src/neutron/neutron/tests/etc/api-paste.ini.test
2014-12-19 20:04:23,313 WARNING [neutron.quota] security_group is already registered.
2014-12-19 20:04:23,314 WARNING [neutron.quota] security_group_rule is already registered.
2014-12-19 20:04:23,322 WARNING [neutron.quota] router is already registered.
2014-12-19 20:04:23,323 WARNING [neutron.quota] floatingip is already registered.
2014-12-19 20:04:24,055 WARNING [gbp.neutron.services.grouppolicy.drivers.resource_mapping] Security Group already exists Security group rule already exists. Rule id is %(id)s.
2014-12-19 20:04:24,069 WARNING [gbp.neutron.services.grouppolicy.drivers.resource_mapping] Security Group already exists Security group rule already exists. Rule id is %(id)s.
2014-12-19 20:04:24,508 WARNING [neutron.scheduler.l3_agent_scheduler] No active L3 agents
2014-12-19 20:04:24,611 WARNING [gbp.neutron.services.grouppolicy.drivers.resource_mapping] Security Group already exists Security group rule already exists. Rule id is %(id)s.
2014-12-19 20:04:24,639 WARNING [gbp.neutron.services.grouppolicy.drivers.resource_mapping] Security Group already exists Security group rule already exists. Rule id is %(id)s.
2014-12-19 20:04:25,075 WARNING [neutron.scheduler.l3_agent_scheduler] No active L3 agents
2014-12-19 20:04:25,198 WARNING [gbp.neutron.services.grouppolicy.drivers.resource_mapping] Security Group already exists Security group rule already exists. Rule id is %(id)s.
2014-12-19 20:04:25,217 WARNING [gbp.neutron.services.grouppolicy.drivers.resource_mapping] Security Group already exists Security group rule already exists. Rule id is %(id)s.
}}}

Traceback (most recent call last):
  File "gbp/neutron/tests/unit/services/grouppolicy/test_resource_mapping.py", line 1712, in test_update_policy_classifier_hier
    self._verify_prs_rules(prs['id'])
  File "gbp/neutron/tests/unit/services/grouppolicy/test_resource_mapping.py", line 273, in _verify_prs_rules
    "Rule not found, expected:\n%s" % rule)
  File "/home/jenkins/workspace/gate-group-based-policy-python27/.tox/py27/local/lib/python2.7/site-packages/unittest2/case.py", line 678, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Rule not found, expected:
{'direction': ['ingress'], 'protocol': [u'tcp'], 'remote_ip_prefix': [u'10.0.0.64/26'], 'port_range_max': [8080], 'security_group_id': [u'1f59175f-46e7-4702-b23e-ffb30d2218bf'], 'port_range_min': [8080]}

Also see the logs here:
http://logs.openstack.org/29/143029/3/check/gate-group-based-policy-python27/7b97432/testr_results.html.gz

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to group-based-policy (master)

Fix proposed to branch: master
Review: https://review.openstack.org/143220

Changed in group-based-policy:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to group-based-policy (master)

Reviewed: https://review.openstack.org/143220
Committed: https://git.openstack.org/cgit/stackforge/group-based-policy/commit/?id=7480dbb60f60f8695b674a632de21802d0f8b9fb
Submitter: Jenkins
Branch: master

commit 7480dbb60f60f8695b674a632de21802d0f8b9fb
Author: Sumit Naiksatam <email address hidden>
Date: Fri Dec 19 15:52:06 2014 -0800

    Temporary work around for a failing UT

    There is some clean up which does not seem to be happening
    properly. Changing the sequence of tests seem to fix this issue
    locally at least.

    Change-Id: I626c68bc32479bcbc2d71007a177fede21824e59
    Closes-bug: 1404412

Changed in group-based-policy:
status: In Progress → Fix Committed
Revision history for this message
Sumit Naiksatam (snaiksat) wrote : Re: Resource Mapping driver UT breaking gate

Reopening this since the issue was not fixed.

Changed in group-based-policy:
status: Fix Committed → Confirmed
assignee: Sumit Naiksatam (snaiksat) → Ivar Lazzaro (mmaleckk)
Changed in group-based-policy:
status: Confirmed → In Progress
Changed in group-based-policy:
milestone: 2014.2rc2 → kilo-gbp-1
Changed in group-based-policy:
importance: Critical → High
Changed in group-based-policy:
milestone: kilo-gbp-1 → kilo-gbp-2
Changed in group-based-policy:
milestone: kilo-gbp-2 → kilo-gbp-3
Ivar Lazzaro (mmaleckk)
summary: - Resource Mapping driver UT breaking gate
+ Session's persistent objects across transactions create inconsistency
Ivar Lazzaro (mmaleckk)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to group-based-policy (master)

Fix proposed to branch: master
Review: https://review.openstack.org/213362

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to group-based-policy (master)

Reviewed: https://review.openstack.org/213362
Committed: https://git.openstack.org/cgit/stackforge/group-based-policy/commit/?id=2c8b401b8e0432bba4f7ae089883b36d760f733c
Submitter: Jenkins
Branch: master

commit 2c8b401b8e0432bba4f7ae089883b36d760f733c
Author: Ivar Lazzaro <email address hidden>
Date: Fri Aug 14 15:36:54 2015 -0700

    expunge session objects in RMD when calling Neutron plugins

    Change-Id: Ic9ccdfbf06cf1c5e1ca1cc648eedce2d09c76540
    Closes-bug: 1404412

Changed in group-based-policy:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to group-based-policy (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/213564

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to group-based-policy (stable/juno)

Reviewed: https://review.openstack.org/213564
Committed: https://git.openstack.org/cgit/stackforge/group-based-policy/commit/?id=85f473ed0f8d4e07afaa1ad08ede943a6e88e98b
Submitter: Jenkins
Branch: stable/juno

commit 85f473ed0f8d4e07afaa1ad08ede943a6e88e98b
Author: Sumit Naiksatam <email address hidden>
Date: Fri Aug 14 15:36:54 2015 -0700

    expunge session objects in RMD when calling Neutron plugins

    Change-Id: Ic9ccdfbf06cf1c5e1ca1cc648eedce2d09c76540
    Closes-bug: 1404412
    (cherry picked from commit 2c8b401b8e0432bba4f7ae089883b36d760f733c)

tags: added: in-stable-juno
Changed in group-based-policy:
milestone: kilo-gbp-3 → kilo-gbp-4
Changed in group-based-policy:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on group-based-policy (master)

Change abandoned by Ivar Lazzaro (<email address hidden>) on branch: master
Review: https://review.openstack.org/144816
Reason: Fixed by https://git.openstack.org/cgit/stackforge/group-based-policy/commit/?id=85f473ed0f8d4e07afaa1ad08ede943a6e88e98b

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.