Ubuntu
neutron package

random sqlite related failure during unit testing

Bug #1676712 reported by James Page
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron (Ubuntu)
Triaged
Medium
Unassigned
vmware-nsx (Ubuntu)
Triaged
Medium
Unassigned

Bug Description

Example:

vmware_nsx.tests.unit.services.qos.test_nsxv3_notification.TestQosNsxV3Notification.test_policy_delete_profile
--------------------------------------------------------------------------------------------------------------

Captured traceback-1:
~~~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/fixtures/fixture.py", line 125, in cleanUp
        return self._cleanups(raise_errors=raise_first)
      File "/usr/lib/python2.7/dist-packages/fixtures/callmany.py", line 89, in __call__
        reraise(error[0], error[1], error[2])
      File "/usr/lib/python2.7/dist-packages/fixtures/callmany.py", line 83, in __call__
        cleanup(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/neutron/tests/unit/testlib_api.py", line 98, in <lambda>
        self.addCleanup(lambda: self._delete_from_schema(engine))
      File "/usr/lib/python2.7/dist-packages/neutron/tests/unit/testlib_api.py", line 83, in _delete_from_schema
        conn.execute(table.delete())
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb, cause=cause)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: nsxv_lbaas_l7policy_bindings [SQL: u'DELETE FROM nsxv_lbaas_l7policy_bindings']

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "vmware_nsx/tests/unit/services/qos/test_nsxv3_notification.py", line 44, in setUp
        super(TestQosNsxV3Notification, self).setUp()
      File "/usr/lib/python2.7/dist-packages/neutron/tests/unit/services/qos/base.py", line 23, in setUp
        super(BaseQosTestCase, self).setUp()
      File "vmware_nsx/tests/unit/nsx_v3/test_plugin.py", line 170, in setUp
        ext_mgr=ext_mgr)
      File "/usr/lib/python2.7/dist-packages/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 132, in setUp
        self.api = router.APIRouter()
      File "/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py", line 77, in __init__
        manager.init()
      File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 291, in init
        NeutronManager.get_instance()
      File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 241, in get_instance
        cls._create_instance()
      File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
        return f(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 227, in _create_instance
        cls._instance = cls()
      File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 131, in __init__
        plugin_provider)
      File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 164, in _get_plugin_instance
        return plugin_class()
      File "/usr/lib/python2.7/dist-packages/neutron/quota/resource_registry.py", line 127, in wrapper
        return f(*args, **kwargs)
      File "vmware_nsx/plugins/nsx_v3/plugin.py", line 205, in __init__
        self._process_security_group_logging()
      File "vmware_nsx/plugins/nsx_v3/plugin.py", line 479, in _process_security_group_logging
        utils.spawn_n(process_security_group_logging)
      File "/usr/lib/python2.7/dist-packages/mock/mock.py", line 1062, in __call__
        return _mock_self._mock_call(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/mock/mock.py", line 1128, in _mock_call
        ret_val = effect(*args, **kwargs)
      File "vmware_nsx/tests/unit/nsx_v/test_plugin.py", line 3298, in <lambda>
        c_utils.spawn_n = mock.Mock(side_effect=lambda f: f())
      File "vmware_nsx/plugins/nsx_v3/plugin.py", line 466, in process_security_group_logging
        sg_logging.LOGGING])
      File "vmware_nsx/plugins/nsx_v3/plugin.py", line 3171, in get_security_groups
        default_sg=default_sg)
      File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 166, in wrapped
        return method(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 95, in wrapped
        setattr(e, '_RETRY_EXCEEDED', True)
      File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
        return f(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
        ectxt.value = e.inner_exc
      File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
        return f(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 131, in wrapped
        traceback.format_exc())
      File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
        return f(*dup_args, **dup_kwargs)
      File "/usr/lib/python2.7/dist-packages/neutron/db/securitygroups_db.py", line 157, in get_security_groups
        page_reverse=page_reverse)
      File "/usr/lib/python2.7/dist-packages/neutron/db/common_db_mixin.py", line 276, in _get_collection
        for c in query]
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
        return self._execute_and_instances(context)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2776, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb, cause=cause)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: securitygroups [SQL: u'SELECT securitygroups.project_id AS securitygroups_project_id, securitygroups.id AS securitygroups_id, securitygroups.name AS securitygroups_name, securitygroups.standard_attr_id AS securitygroups_standard_attr_id, standardattributes_1.id AS standardattributes_1_id, standardattributes_1.resource_type AS standardattributes_1_resource_type, standardattributes_1.description AS standardattributes_1_description, standardattributes_1.revision_number AS standardattributes_1_revision_number, standardattributes_1.created_at AS standardattributes_1_created_at, standardattributes_1.updated_at AS standardattributes_1_updated_at, nsx_extended_security_group_properties_1.security_group_id AS nsx_extended_security_group_properties_1_security_group_id, nsx_extended_security_group_properties_1.logging AS nsx_extended_security_group_properties_1_logging, nsx_extended_security_group_properties_1.provider AS nsx_extended_security_group_properties_1_provider, nsx_extended_security_group_properties_1.policy AS nsx_extended_security_group_properties_1_policy \nFROM securitygroups LEFT OUTER JOIN standardattributes AS standardattributes_1 ON standardattributes_1.id = securitygroups.standard_attr_id LEFT OUTER JOIN nsx_extended_security_group_properties AS nsx_extended_security_group_properties_1 ON securitygroups.id = nsx_extended_security_group_properties_1.security_group_id']

James Page (james-page)
Changed in vmware-nsx (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
James Page (james-page) wrote :

I've also seen the same issue (but less frequently) with neutron.

Seems to occur mostly frequently in Launchpad builders which are 4 Core, 4GB RAM, 4GB swap.

Relatively easy to reproduce on an instance of the same spec but running two instances of the same build in parallel using sbuild.

Changed in neutron (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Arjun Baindur (abaindur) wrote :

We see this frequently with neutron, any update on root cause?

Revision history for this message
LIU Yulong (dragon889) wrote :

Meets the same issue in our jenkins CI ut job.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This seems to be occurring more frequently and is getting painful. We might consider running tests sequentially.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.