Snapshots fail: Failed to add object to Swift

Bug #1921893 reported by Oliver
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Glance
New
Undecided
Unassigned

Bug Description

I'm using OpenStack Victoria.

When I try to take snapshots of instances above a certain size, they disappear during saving state and glance shows the following error:

Mar 30 09:15:14 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:15:14.306 96 INFO glance.api.v2.image_data [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.
Mar 30 09:16:10 infra1-glance-container-99614ac2 uwsgi[96]: Tue Mar 30 09:16:10 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2/images/c83f6c4d-0083-4600-95f5-d21b4623af47/file (ip 192.168.110.214) !!!
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.427 96 ERROR glance_store._drivers.swift.store [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.494 96 ERROR glance_store._drivers.swift.store [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift.
                                                                      Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.497 96 ERROR glance.api.v2.image_data [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift.
Mar 30 09:16:11 infra1-glance-container-99614ac2 glance-wsgi-api[96]: 2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift.
                                                                      Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift.
                                                                      Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call last):
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._delete_stale_chunks(
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi manager.get_connection().put_object(
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents,
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi reset_func(func, *args, **kwargs)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no '
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload.
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi During handling of the above exception, another exception occurred:
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Traceback (most recent call last):
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return method(*args, **kwargs)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._restore(image_repo, image)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.force_reraise()
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise value
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend,
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend,
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash(
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return store_add_to_backend_with_multihash(
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add(
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi raise glance_store.BackendException(msg)
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift.
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', 'c83f6c4d-0083-4600-95f5-d21b4623af47-00002', ...) failure and no ability to reset contents for reupload..
                                                                      2021-03-30 09:16:11.600 96 ERROR glance.common.wsgi

I don't see errors in the swift logs:

Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0135 RL - 1617095714.876815081 1617095714.890305758 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0131 CS - 1617095714.892540216 1617095714.905595303 -
Mar 30 09:15:14 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: - - 30/Mar/2021/09/15/14 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 204 - Swift - - - - txf319964369254af4974a7-006062ec22 - 0.0127 CS - 1617095714.907815695 1617095714.920493126 0
Mar 30 09:15:15 infra1-swift-proxy-container-27169fa7 proxy-server[1068]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/15/15 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - - txf319964369254af4974a7-006062ec22 - 0.2779 - - 1617095714.874005318 1617095715.151889324 0
Mar 30 09:15:17 infra1-swift-proxy-container-27169fa7 proxy-server[1071]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/15/17 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001 HTTP/1.0 201 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 - - txc05fff07e3384a68a7fa9-006062ec23 - 2.8218 - - 1617095715.163244247 1617095717.985031843 0
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]: Client disconnected without sending last chunk (txn: tx49b0efe394364bf1975bb-006062ec25) (client_ip: 192.168.110.106)
Mar 30 09:16:10 infra1-swift-proxy-container-27169fa7 proxy-server[1067]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/16/10 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002 HTTP/1.0 499 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... 204800000 89 - tx49b0efe394364bf1975bb-006062ec25 - 52.2224 - - 1617095717.995234489 1617095770.217619896 0
Mar 30 09:16:11 infra1-swift-proxy-container-27169fa7 proxy-server[1080]: 192.168.110.106 192.168.110.211 30/Mar/2021/09/16/11 DELETE /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001 HTTP/1.0 204 - python-swiftclient-3.10.1 gAAAAABgYuwin4R-... - - - tx98a9247419a2445fafcbf-006062ec5b - 0.0486 - - 1617095771.441193104 1617095771.489775896 0

and

Mar 30 09:15:17 bc1bl12 object-server[694851]: 10.0.3.211 - - [30/Mar/2021:09:15:17 +0000] "PUT /os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" 201 - "PUT http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" "txc05fff07e3384a68a7fa9-006062ec23" "proxy-server 1071" 2.8030 "-" 694851 0
Mar 30 09:16:10 bc1bl12 object-server[694850]: 10.0.3.211 - - [30/Mar/2021:09:16:10 +0000] "PUT /os-objects/43/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002" 499 89 "PUT http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00002" "tx49b0efe394364bf1975bb-006062ec25" "proxy-server 1067" 52.1551 "-" 694850 0
Mar 30 09:16:11 bc1bl12 object-server[694854]: 10.0.3.211 - - [30/Mar/2021:09:16:11 +0000] "DELETE /os-objects/245/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" 204 - "DELETE http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images/c83f6c4d-0083-4600-95f5-d21b4623af47-00001" "tx98a9247419a2445fafcbf-006062ec5b" "proxy-server 1080" 0.0321 "-" 694854 0

nova-compute shows the following:

Mar 30 09:13:24 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:24.975 650019 INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] instance snapshotting
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.038 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Beginning live snapshot process
Mar 30 09:13:25 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:25.527 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Skipping quiescing instance: QEMU guest agent is not enabled.
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.384 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.386 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:13:56 bc1bl14 nova-compute[650019]: 2021-03-30 09:13:56.983 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.390 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:14:57 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:57.391 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:14:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:14:58.006 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:15:13 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:13.861 650019 INFO nova.virt.libvirt.driver [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Snapshot extracted, beginning image upload
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.395 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.397 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:15:58 bc1bl14 nova-compute[650019]: 2021-03-30 09:15:58.997 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.530 650019 INFO nova.compute.manager [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted task state from image_uploading on failure for instance.
Mar 30 09:16:12 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server [req-551ffa80-a1c6-462d-931a-84069b23995a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Exception during message handling: glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2478, in snapshot
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot(
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py", line 452, in direct_snapshot
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is not implemented'))
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not implemented
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 76, in wrapped
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server _emit_exception_notification(
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 69, in wrapped
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 188, in decorated_function
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 159, in decorated_function
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 216, in decorated_function
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 205, in decorated_function
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 236, in decorated_function
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server compute_utils.delete_image(
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.force_reraise()
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 232, in decorated_function
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance,
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3908, in snapshot_instance
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance,
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3941, in _snapshot_instance
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id,
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2549, in snapshot
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server self._image_api.update(context,
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1247, in update
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return session.update(context, image_id, image_info, data=data,
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 696, in update
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1035, in _reraise_translated_image_exception
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 694, in update
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 713, in _update_v2
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server image = self._upload_data(context, image_id, data)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 589, in _upload_data
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call,
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line 694, in tpool_execute
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise value
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 192, in call
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py", line 600, in inner
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py", line 289, in upload
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url, headers=hdrs, data=body)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 380, in request
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server return self._handle_response(resp)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 120, in _handle_response
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
                                              2021-03-30 09:16:12.533 650019 ERROR oslo_messaging.rpc.server
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.391 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.392 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking
Mar 30 09:16:59 bc1bl14 nova-compute[650019]: 2021-03-30 09:16:59.997 650019 INFO nova.virt.libvirt.imagecache [req-9c41b929-15ea-4228-b648-ddad2bcd6e13 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3

Revision history for this message
Adrien Cunin (adri2000) wrote :

We experience something similar in Xena.

Revision history for this message
alpha23 (alpha23) wrote :

Any update on this issue? Experiencing the same in Xena.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.