| 2016-09-15 12:54:29 |
Timur Nurlygayanov |
bug |
|
|
added bug |
| 2016-09-15 12:54:38 |
Timur Nurlygayanov |
fuel: assignee |
|
MOS Cinder (mos-cinder) |
|
| 2016-09-15 12:58:04 |
Timur Nurlygayanov |
attachment added |
|
fuel-snapshot-2016-09-15_12-34-19.tar https://bugs.launchpad.net/fuel/+bug/1623934/+attachment/4741404/+files/fuel-snapshot-2016-09-15_12-34-19.tar |
|
| 2016-09-15 13:01:16 |
Timur Nurlygayanov |
description |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
all object-storage OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information. |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
all object-storage OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information.
In Cinder logs we can see the error in MySQL database for cinder:
2016-09-15 10:37:04.799 30924 DEBUG oslo_db.sqlalchemy.engines [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
2016-09-15 10:37:05.089 30924 CRITICAL cinder [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder Traceback (most recent call last):
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/bin/cinder-api", line 10, in <module>
2016-09-15 10:37:05.089 30924 ERROR cinder sys.exit(main())
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/api.py", line 61, in main
2016-09-15 10:37:05.089 30924 ERROR cinder server = service.WSGIService('osapi_volume')
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 375, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.app = self.loader.load_app(name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
2016-09-15 10:37:05.089 30924 ERROR cinder return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-09-15 10:37:05.089 30924 ERROR cinder return loadobj(APP, uri, name=name, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-09-15 10:37:05.089 30924 ERROR cinder return context.create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/__init__.py", line 34, in root_app_factory
2016-09-15 10:37:05.089 30924 ERROR cinder return paste.urlmap.urlmap_factory(loader, global_conf, **local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(app_name, global_conf=global_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/auth.py", line 56, in pipeline_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(pipeline[-1])
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder return fix_call(context.object, context.global_conf, **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 72, in factory
2016-09-15 10:37:05.089 30924 ERROR cinder return cls()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 83, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self._setup_routes(mapper, ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/router.py", line 50, in _setup_routes
2016-09-15 10:37:05.089 30924 ERROR cinder self.resources['volumes'] = volumes.create_resource(ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 473, in create_resource
2016-09-15 10:37:05.089 30924 ERROR cinder return wsgi.Resource(VolumeController(ext_mgr))
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 165, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.volume_api = cinder_volume.API()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/__init__.py", line 30, in API
2016-09-15 10:37:05.089 30924 ERROR cinder return importutils.import_object(class_name, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2016-09-15 10:37:05.089 30924 ERROR cinder return import_class(import_str)(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 130, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 187, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder obj_version_cap = self._determine_obj_version_cap()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 221, in _determine_obj_version_cap
2016-09-15 10:37:05.089 30924 ERROR cinder cinder.context.get_admin_context(), self.BINARY)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 133, in get_minimum_obj_version
2016-09-15 10:37:05.089 30924 ERROR cinder binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 107, in _get_minimum_version
2016-09-15 10:37:05.089 30924 ERROR cinder services = ServiceList.get_all_by_binary(context, binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 162, in get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder disabled=disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 122, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return IMPL.service_get_all_by_binary(context, binary, disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 175, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder return f(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 429, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return query.all()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
2016-09-15 10:37:05.089 30924 ERROR cinder return list(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-09-15 10:37:05.089 30924 ERROR cinder return self._execute_and_instances(context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2016-09-15 10:37:05.089 30924 ERROR cinder result = conn.execute(querycontext.statement, self._params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder return meth(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-09-15 10:37:05.089 30924 ERROR cinder return connection._execute_clauseelement(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-09-15 10:37:05.089 30924 ERROR cinder compiled_sql, distilled_params
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-09-15 10:37:05.089 30924 ERROR cinder util.raise_from_cause(newraise, exc_info)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-09-15 10:37:05.089 30924 ERROR cinder reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-09-15 10:37:05.089 30924 ERROR cinder cursor.execute(statement, parameters)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder self.errorhandler(self, exc, value)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler
2016-09-15 10:37:05.089 30924 ERROR cinder raise errorvalue
2016-09-15 10:37:05.089 30924 ERROR cinder ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder |
|
| 2016-09-15 13:07:18 |
Timur Nurlygayanov |
tags |
|
blocker-for-qa tempest |
|
| 2016-09-15 13:07:31 |
Timur Nurlygayanov |
tags |
blocker-for-qa tempest |
area-cinder blocker-for-qa tempest |
|
| 2016-09-15 13:14:05 |
Ivan Kolodyazhny |
fuel: assignee |
MOS Cinder (mos-cinder) |
Ivan Kolodyazhny (e0ne) |
|
| 2016-09-15 13:15:25 |
Timur Nurlygayanov |
description |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
all object-storage OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information.
In Cinder logs we can see the error in MySQL database for cinder:
2016-09-15 10:37:04.799 30924 DEBUG oslo_db.sqlalchemy.engines [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
2016-09-15 10:37:05.089 30924 CRITICAL cinder [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder Traceback (most recent call last):
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/bin/cinder-api", line 10, in <module>
2016-09-15 10:37:05.089 30924 ERROR cinder sys.exit(main())
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/api.py", line 61, in main
2016-09-15 10:37:05.089 30924 ERROR cinder server = service.WSGIService('osapi_volume')
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 375, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.app = self.loader.load_app(name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
2016-09-15 10:37:05.089 30924 ERROR cinder return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-09-15 10:37:05.089 30924 ERROR cinder return loadobj(APP, uri, name=name, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-09-15 10:37:05.089 30924 ERROR cinder return context.create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/__init__.py", line 34, in root_app_factory
2016-09-15 10:37:05.089 30924 ERROR cinder return paste.urlmap.urlmap_factory(loader, global_conf, **local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(app_name, global_conf=global_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/auth.py", line 56, in pipeline_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(pipeline[-1])
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder return fix_call(context.object, context.global_conf, **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 72, in factory
2016-09-15 10:37:05.089 30924 ERROR cinder return cls()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 83, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self._setup_routes(mapper, ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/router.py", line 50, in _setup_routes
2016-09-15 10:37:05.089 30924 ERROR cinder self.resources['volumes'] = volumes.create_resource(ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 473, in create_resource
2016-09-15 10:37:05.089 30924 ERROR cinder return wsgi.Resource(VolumeController(ext_mgr))
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 165, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.volume_api = cinder_volume.API()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/__init__.py", line 30, in API
2016-09-15 10:37:05.089 30924 ERROR cinder return importutils.import_object(class_name, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2016-09-15 10:37:05.089 30924 ERROR cinder return import_class(import_str)(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 130, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 187, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder obj_version_cap = self._determine_obj_version_cap()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 221, in _determine_obj_version_cap
2016-09-15 10:37:05.089 30924 ERROR cinder cinder.context.get_admin_context(), self.BINARY)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 133, in get_minimum_obj_version
2016-09-15 10:37:05.089 30924 ERROR cinder binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 107, in _get_minimum_version
2016-09-15 10:37:05.089 30924 ERROR cinder services = ServiceList.get_all_by_binary(context, binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 162, in get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder disabled=disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 122, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return IMPL.service_get_all_by_binary(context, binary, disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 175, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder return f(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 429, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return query.all()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
2016-09-15 10:37:05.089 30924 ERROR cinder return list(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-09-15 10:37:05.089 30924 ERROR cinder return self._execute_and_instances(context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2016-09-15 10:37:05.089 30924 ERROR cinder result = conn.execute(querycontext.statement, self._params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder return meth(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-09-15 10:37:05.089 30924 ERROR cinder return connection._execute_clauseelement(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-09-15 10:37:05.089 30924 ERROR cinder compiled_sql, distilled_params
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-09-15 10:37:05.089 30924 ERROR cinder util.raise_from_cause(newraise, exc_info)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-09-15 10:37:05.089 30924 ERROR cinder reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-09-15 10:37:05.089 30924 ERROR cinder cursor.execute(statement, parameters)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder self.errorhandler(self, exc, value)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler
2016-09-15 10:37:05.089 30924 ERROR cinder raise errorvalue
2016-09-15 10:37:05.089 30924 ERROR cinder ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
All volume-related OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information.
In Cinder logs we can see the error in MySQL database for cinder:
2016-09-15 10:37:04.799 30924 DEBUG oslo_db.sqlalchemy.engines [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
2016-09-15 10:37:05.089 30924 CRITICAL cinder [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder Traceback (most recent call last):
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/bin/cinder-api", line 10, in <module>
2016-09-15 10:37:05.089 30924 ERROR cinder sys.exit(main())
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/api.py", line 61, in main
2016-09-15 10:37:05.089 30924 ERROR cinder server = service.WSGIService('osapi_volume')
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 375, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.app = self.loader.load_app(name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
2016-09-15 10:37:05.089 30924 ERROR cinder return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-09-15 10:37:05.089 30924 ERROR cinder return loadobj(APP, uri, name=name, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-09-15 10:37:05.089 30924 ERROR cinder return context.create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/__init__.py", line 34, in root_app_factory
2016-09-15 10:37:05.089 30924 ERROR cinder return paste.urlmap.urlmap_factory(loader, global_conf, **local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(app_name, global_conf=global_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/auth.py", line 56, in pipeline_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(pipeline[-1])
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder return fix_call(context.object, context.global_conf, **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 72, in factory
2016-09-15 10:37:05.089 30924 ERROR cinder return cls()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 83, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self._setup_routes(mapper, ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/router.py", line 50, in _setup_routes
2016-09-15 10:37:05.089 30924 ERROR cinder self.resources['volumes'] = volumes.create_resource(ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 473, in create_resource
2016-09-15 10:37:05.089 30924 ERROR cinder return wsgi.Resource(VolumeController(ext_mgr))
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 165, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.volume_api = cinder_volume.API()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/__init__.py", line 30, in API
2016-09-15 10:37:05.089 30924 ERROR cinder return importutils.import_object(class_name, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2016-09-15 10:37:05.089 30924 ERROR cinder return import_class(import_str)(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 130, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 187, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder obj_version_cap = self._determine_obj_version_cap()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 221, in _determine_obj_version_cap
2016-09-15 10:37:05.089 30924 ERROR cinder cinder.context.get_admin_context(), self.BINARY)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 133, in get_minimum_obj_version
2016-09-15 10:37:05.089 30924 ERROR cinder binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 107, in _get_minimum_version
2016-09-15 10:37:05.089 30924 ERROR cinder services = ServiceList.get_all_by_binary(context, binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 162, in get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder disabled=disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 122, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return IMPL.service_get_all_by_binary(context, binary, disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 175, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder return f(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 429, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return query.all()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
2016-09-15 10:37:05.089 30924 ERROR cinder return list(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-09-15 10:37:05.089 30924 ERROR cinder return self._execute_and_instances(context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2016-09-15 10:37:05.089 30924 ERROR cinder result = conn.execute(querycontext.statement, self._params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder return meth(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-09-15 10:37:05.089 30924 ERROR cinder return connection._execute_clauseelement(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-09-15 10:37:05.089 30924 ERROR cinder compiled_sql, distilled_params
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-09-15 10:37:05.089 30924 ERROR cinder util.raise_from_cause(newraise, exc_info)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-09-15 10:37:05.089 30924 ERROR cinder reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-09-15 10:37:05.089 30924 ERROR cinder cursor.execute(statement, parameters)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder self.errorhandler(self, exc, value)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler
2016-09-15 10:37:05.089 30924 ERROR cinder raise errorvalue
2016-09-15 10:37:05.089 30924 ERROR cinder ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder |
|
| 2016-09-15 13:15:35 |
Timur Nurlygayanov |
nominated for series |
|
fuel/mitaka |
|
| 2016-09-15 13:15:35 |
Timur Nurlygayanov |
bug task added |
|
fuel/mitaka |
|
| 2016-09-15 13:15:35 |
Timur Nurlygayanov |
nominated for series |
|
fuel/newton |
|
| 2016-09-15 13:15:35 |
Timur Nurlygayanov |
bug task added |
|
fuel/newton |
|
| 2016-09-15 13:15:50 |
Timur Nurlygayanov |
fuel/mitaka: importance |
Undecided |
High |
|
| 2016-09-15 13:15:52 |
Timur Nurlygayanov |
fuel/newton: importance |
Undecided |
High |
|
| 2016-09-15 13:16:02 |
Timur Nurlygayanov |
fuel/mitaka: assignee |
|
Ivan Kolodyazhny (e0ne) |
|
| 2016-09-15 13:16:05 |
Timur Nurlygayanov |
fuel/mitaka: milestone |
|
9.1 |
|
| 2016-09-15 13:16:07 |
Timur Nurlygayanov |
fuel/newton: milestone |
|
10.0 |
|
| 2016-09-15 13:16:10 |
Timur Nurlygayanov |
fuel/mitaka: status |
New |
Confirmed |
|
| 2016-09-15 13:16:12 |
Timur Nurlygayanov |
fuel/newton: status |
New |
Confirmed |
|
| 2016-09-15 13:30:20 |
Timur Nurlygayanov |
summary |
OSTF tests for cinder failed on configuration with Cinder LVM, Sahara, Ceilometer and Ironic |
OSTF tests for Nova failed on configuration with Cinder LVM, Sahara, Ceilometer and Ironic |
|
| 2016-09-15 13:31:23 |
Ivan Kolodyazhny |
fuel/mitaka: assignee |
Ivan Kolodyazhny (e0ne) |
MOS Nova (mos-nova) |
|
| 2016-09-15 13:31:41 |
Ivan Kolodyazhny |
fuel/newton: assignee |
Ivan Kolodyazhny (e0ne) |
MOS Nova (mos-nova) |
|
| 2016-09-15 13:38:45 |
Timur Nurlygayanov |
description |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
All volume-related OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information.
In Cinder logs we can see the error in MySQL database for cinder:
2016-09-15 10:37:04.799 30924 DEBUG oslo_db.sqlalchemy.engines [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
2016-09-15 10:37:05.089 30924 CRITICAL cinder [req-ef8c5f39-dae3-47b1-a182-fa1a26eabbfc - - - - -] ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder Traceback (most recent call last):
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/bin/cinder-api", line 10, in <module>
2016-09-15 10:37:05.089 30924 ERROR cinder sys.exit(main())
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/api.py", line 61, in main
2016-09-15 10:37:05.089 30924 ERROR cinder server = service.WSGIService('osapi_volume')
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 375, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.app = self.loader.load_app(name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
2016-09-15 10:37:05.089 30924 ERROR cinder return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-09-15 10:37:05.089 30924 ERROR cinder return loadobj(APP, uri, name=name, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-09-15 10:37:05.089 30924 ERROR cinder return context.create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/__init__.py", line 34, in root_app_factory
2016-09-15 10:37:05.089 30924 ERROR cinder return paste.urlmap.urlmap_factory(loader, global_conf, **local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(app_name, global_conf=global_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/auth.py", line 56, in pipeline_factory
2016-09-15 10:37:05.089 30924 ERROR cinder app = loader.get_app(pipeline[-1])
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-09-15 10:37:05.089 30924 ERROR cinder name=name, global_conf=global_conf).create()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-15 10:37:05.089 30924 ERROR cinder return self.object_type.invoke(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-09-15 10:37:05.089 30924 ERROR cinder return fix_call(context.object, context.global_conf, **context.local_conf)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-15 10:37:05.089 30924 ERROR cinder val = callable(*args, **kw)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 72, in factory
2016-09-15 10:37:05.089 30924 ERROR cinder return cls()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/__init__.py", line 83, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self._setup_routes(mapper, ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/router.py", line 50, in _setup_routes
2016-09-15 10:37:05.089 30924 ERROR cinder self.resources['volumes'] = volumes.create_resource(ext_mgr)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 473, in create_resource
2016-09-15 10:37:05.089 30924 ERROR cinder return wsgi.Resource(VolumeController(ext_mgr))
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/api/v2/volumes.py", line 165, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.volume_api = cinder_volume.API()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/__init__.py", line 30, in API
2016-09-15 10:37:05.089 30924 ERROR cinder return importutils.import_object(class_name, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2016-09-15 10:37:05.089 30924 ERROR cinder return import_class(import_str)(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 130, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 187, in __init__
2016-09-15 10:37:05.089 30924 ERROR cinder obj_version_cap = self._determine_obj_version_cap()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 221, in _determine_obj_version_cap
2016-09-15 10:37:05.089 30924 ERROR cinder cinder.context.get_admin_context(), self.BINARY)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 133, in get_minimum_obj_version
2016-09-15 10:37:05.089 30924 ERROR cinder binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 107, in _get_minimum_version
2016-09-15 10:37:05.089 30924 ERROR cinder services = ServiceList.get_all_by_binary(context, binary)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder result = fn(cls, context, *args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 162, in get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder disabled=disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 122, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return IMPL.service_get_all_by_binary(context, binary, disabled)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 175, in wrapper
2016-09-15 10:37:05.089 30924 ERROR cinder return f(*args, **kwargs)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 429, in service_get_all_by_binary
2016-09-15 10:37:05.089 30924 ERROR cinder return query.all()
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
2016-09-15 10:37:05.089 30924 ERROR cinder return list(self)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-09-15 10:37:05.089 30924 ERROR cinder return self._execute_and_instances(context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2016-09-15 10:37:05.089 30924 ERROR cinder result = conn.execute(querycontext.statement, self._params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder return meth(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-09-15 10:37:05.089 30924 ERROR cinder return connection._execute_clauseelement(self, multiparams, params)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-09-15 10:37:05.089 30924 ERROR cinder compiled_sql, distilled_params
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-09-15 10:37:05.089 30924 ERROR cinder util.raise_from_cause(newraise, exc_info)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-09-15 10:37:05.089 30924 ERROR cinder reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-09-15 10:37:05.089 30924 ERROR cinder context)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-09-15 10:37:05.089 30924 ERROR cinder cursor.execute(statement, parameters)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
2016-09-15 10:37:05.089 30924 ERROR cinder self.errorhandler(self, exc, value)
2016-09-15 10:37:05.089 30924 ERROR cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler
2016-09-15 10:37:05.089 30924 ERROR cinder raise errorvalue
2016-09-15 10:37:05.089 30924 ERROR cinder ProgrammingError: (_mysql_exceptions.ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %s'] [parameters: ('cinder-scheduler',)]
2016-09-15 10:37:05.089 30924 ERROR cinder |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
All volume-related OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information. |
|
| 2016-09-15 13:38:57 |
Timur Nurlygayanov |
description |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
All volume-related OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information. |
Note:
The issue reproduced on Tempest CI, so, this is a regression issue.
Steps To Reproduce:
1) Deploy the following configuration: 3 controllers with mongo, 2 compute node with cinder backend, 2 Ironic nodes
2) Run OSTF tests
Observed Results:
All VM-related OSTF tests failed (boot VM from volume and etc.)
Expected Results:
All OSTF tests passed.
Please see attached diagnostic snapshot for more detailed information. |
|
| 2016-09-15 13:39:10 |
Timur Nurlygayanov |
summary |
OSTF tests for Nova failed on configuration with Cinder LVM, Sahara, Ceilometer and Ironic |
[regression] OSTF tests for Nova failed on configuration with Cinder LVM, Sahara, Ceilometer and Ironic |
|
| 2016-09-15 15:48:24 |
Roman Podoliaka |
fuel/mitaka: assignee |
MOS Nova (mos-nova) |
MOS QA Team (mos-qa) |
|
| 2016-09-15 15:48:34 |
Roman Podoliaka |
fuel/mitaka: assignee |
MOS QA Team (mos-qa) |
Timur Nurlygayanov (tnurlygayanov) |
|
| 2016-09-15 15:48:39 |
Roman Podoliaka |
fuel/newton: assignee |
MOS Nova (mos-nova) |
Timur Nurlygayanov (tnurlygayanov) |
|
| 2016-09-15 15:48:46 |
Roman Podoliaka |
tags |
area-cinder blocker-for-qa tempest |
blocker-for-qa tempest |
|
| 2016-09-15 16:14:28 |
Timur Nurlygayanov |
fuel/mitaka: importance |
High |
Critical |
|
| 2016-09-15 16:14:30 |
Timur Nurlygayanov |
fuel/newton: importance |
High |
Critical |
|
| 2016-09-16 08:54:44 |
Timur Nurlygayanov |
fuel/mitaka: status |
Confirmed |
Invalid |
|
| 2016-09-16 08:54:46 |
Timur Nurlygayanov |
fuel/newton: status |
Confirmed |
Invalid |
|
| 2016-09-16 10:11:54 |
Maksim Malchuk |
bug task deleted |
fuel/newton |
|
|
