Job details:
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1262/
Attempt to snapshot instance fails with:
Glance-api.log:
2017-03-03T14:18:45.979868+00:00 err: 2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters [req-24b7ae06-cf5a-4c77-b2ef-c573bb661bf5 2b8ec16ca29e402396a687c28bf4d83a d2652d0ca5c6433895d785d04adde08
4 - default default] DBAPIError exception wrapped from (pymysql.err.InternalError) (1047, u'WSREP has not yet prepared node for application use')
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters self.engine.dialect.do_rollback(self.connection)
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters dbapi_connection.rollback()
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 772, in rollback
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters self._read_ok_packet()
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 746, in _read_ok_packet
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters pkt = self._read_packet()
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 981, in _read_packet
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in check_error
2017-03-03 14:18:45.911 4297 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
2017-03-03 14:18:45.91
2017-03-03T14:18:45.988778+00:00 info: /usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py:68: SAWarning: An exception has occurred during handling of a previous exception. The previous exception i
s:
2017-03-03T14:18:45.991169+00:00 info: <class 'oslo_db.exception.DBDeadlock'> (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')
2017-03-03T14:18:45.991169+00:00 info: "is:\n %s %s\n" % (self._exc_info[0], self._exc_info[1]))
2017-03-03T14:18:46.103720+00:00 err: 2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data [req-24b7ae06-cf5a-4c77-b2ef-c573bb661bf5 2b8ec16ca29e402396a687c28bf4d83a d2652d0ca5c6433895d785d04adde084 - de
fault default] Failed to upload image data due to internal error
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data Traceback (most recent call last):
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/api/v2/image_data.py", line 118, in upload
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data image_repo.save(image, from_state='saving')
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 99, in save
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data result = self.base.save(base_item, from_state=from_state)
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/notifier.py", line 509, in save
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data super(ImageRepoProxy, self).save(image, from_state=from_state)
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 99, in save
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data result = self.base.save(base_item, from_state=from_state)
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/api/policy.py", line 118, in save
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data return super(ImageRepoProxy, self).save(image, from_state=from_state)
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 99, in save
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data result = self.base.save(base_item, from_state=from_state)
2017-03-03 14:18:45.988 4297 ERROR glance.api.v2.image_data File "/usr/lib/py
2017-03-03T14:18:46.105772+00:00 debug: 2017-03-03 14:18:46.102 4297 DEBUG glance.domain [req-24b7ae06-cf5a-4c77-b2ef-c573bb661bf5 2b8ec16ca29e402396a687c28bf4d83a d2652d0ca5c6433895d785d04adde084 - default def
ault] Image status transition from active to queued is not allowed status /usr/lib/python2.7/dist-packages/glance/domain/__init__.py:148
2017-03-03T14:18:46.124115+00:00 err: 2017-03-03 14:18:46.104 4297 ERROR glance.api.v2.image_data [req-24b7ae06-cf5a-4c77-b2ef-c573bb661bf5 2b8ec16ca29e402396a687c28bf4d83a d2652d0ca5c6433895d785d04adde084 - default default] Unable to restore image b6793317-f30d-4241-9b86-9a2ec71a2e48: Image status transition from active to queued is not allowed
nova-compute.log:
2017-03-03T14:18:46.227720+00:00 debug: 2017-03-03 14:18:46.215 22249 DEBUG nova.compute.manager [req-95969600-43a8-4108-89b3-45e63b8e1882 2b8ec16ca29e402396a687c28bf4d83a d2652d0ca5c6433895d785d04adde084 - - -
] [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] Cleaning up image b6793317-f30d-4241-9b86-9a2ec71a2e48 decorated_function /usr/lib/python2.7/dist-packages/nova/compute/manager.py:236
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] Traceback (most recent call last):
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 232, in decorated_function
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] *args, **kwargs)
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3042, in snapshot_instance
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] task_states.IMAGE_SNAPSHOT)
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3072, in _snapshot_instance
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] update_task_state)
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1565, in snapshot
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] image_file)
2017-03-03 14:18:46.215 22249 ERROR nova.compute.manager [instance: 8b9064e6-08a8-4c14-a45c-651506f91654] File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 130,
Please attach diagnostic snapshot, marking as Incomplete.