Mar 11 19:17:31 cloud-c1-1030989-glance-container-071516d6 uwsgi[32133]: /openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_policy/policy.py:743: UserWarning: Policy "default":"role:admin" was deprecated in Ussuri in favor of "default":"". Reason: In order to allow operators to accept the default policies from code by not defining them in the policy file, while still working with old policy files that rely on the ``default`` rule for policies that are not specified in the policy file, the ``default`` rule must now be explicitly set to ``"role:admin"`` when that is the desired default for unspecified rules.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually. Mar 11 19:17:31 cloud-c1-1030989-glance-container-071516d6 uwsgi[32133]: warnings.warn(deprecated_msg) Mar 11 19:17:31 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32135]: 2021-03-11 19:17:31.701 32135 ERROR glance_store._drivers.rbd [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to store image 0a8a79d0-afe7-402f-b256-80b31d0dea0d Store Exception RBD incomplete write (Wrote only 8388608 out of 8402336 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8402336 bytes) Mar 11 19:17:31 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32135]: 2021-03-11 19:17:31.855 32135 ERROR glance.api.v2.image_data [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to upload image data due to internal error: rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8402336 bytes) Mar 11 19:17:31 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32135]: 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Caught error: RBD incomplete write (Wrote only 8388608 out of 8402336 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8402336 bytes) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi raise value 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi raise value 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 630, in add 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi raise exc 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 586, in add 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi image.write(chunk, offset) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "rbd.pyx", line 3304, in rbd.requires_not_closed.wrapper 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi File "rbd.pyx", line 4393, in rbd.Image.write 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8402336 bytes) 2021-03-11 19:17:31.897 32135 ERROR glance.common.wsgi Mar 11 19:17:32 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32131]: 2021-03-11 19:17:32.953 32131 INFO glance.api.v2.image_data [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 11 19:17:33 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32131]: 2021-03-11 19:17:33.070 32131 WARNING glance_store._drivers.rbd [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Since image size is zero we will be doing resize-before-write which will be slower than normal Mar 11 19:17:33 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32131]: 2021-03-11 19:17:33.192 32131 ERROR glance_store._drivers.rbd [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to store image 0a8a79d0-afe7-402f-b256-80b31d0dea0d Store Exception RBD incomplete write (Wrote only 8388608 out of 8389617 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8389617 bytes) Mar 11 19:17:33 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32131]: 2021-03-11 19:17:33.307 32131 ERROR glance.api.v2.image_data [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to upload image data due to internal error: rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8389617 bytes) Mar 11 19:17:33 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32131]: 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Caught error: RBD incomplete write (Wrote only 8388608 out of 8389617 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8389617 bytes) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi raise value 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi raise value 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 630, in add 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi raise exc 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 586, in add 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi image.write(chunk, offset) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "rbd.pyx", line 3304, in rbd.requires_not_closed.wrapper 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi File "rbd.pyx", line 4393, in rbd.Image.write 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8389617 bytes) 2021-03-11 19:17:33.347 32131 ERROR glance.common.wsgi Mar 11 19:17:34 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32137]: 2021-03-11 19:17:34.451 32137 INFO glance.api.v2.image_data [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 11 19:17:34 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32137]: 2021-03-11 19:17:34.561 32137 WARNING glance_store._drivers.rbd [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Since image size is zero we will be doing resize-before-write which will be slower than normal Mar 11 19:17:34 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32137]: 2021-03-11 19:17:34.680 32137 ERROR glance_store._drivers.rbd [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to store image 0a8a79d0-afe7-402f-b256-80b31d0dea0d Store Exception RBD incomplete write (Wrote only 8388608 out of 8404238 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8404238 bytes) Mar 11 19:17:34 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32137]: 2021-03-11 19:17:34.847 32137 ERROR glance.api.v2.image_data [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to upload image data due to internal error: rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8404238 bytes) Mar 11 19:17:34 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32137]: 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Caught error: RBD incomplete write (Wrote only 8388608 out of 8404238 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8404238 bytes) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi raise value 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi raise value 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 630, in add 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi raise exc 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 586, in add 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi image.write(chunk, offset) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "rbd.pyx", line 3304, in rbd.requires_not_closed.wrapper 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi File "rbd.pyx", line 4393, in rbd.Image.write 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8404238 bytes) 2021-03-11 19:17:34.911 32137 ERROR glance.common.wsgi Mar 11 19:17:35 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32128]: 2021-03-11 19:17:35.986 32128 INFO glance.api.v2.image_data [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 11 19:17:36 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32128]: 2021-03-11 19:17:36.131 32128 WARNING glance_store._drivers.rbd [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Since image size is zero we will be doing resize-before-write which will be slower than normal Mar 11 19:17:36 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32128]: 2021-03-11 19:17:36.269 32128 ERROR glance_store._drivers.rbd [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to store image 0a8a79d0-afe7-402f-b256-80b31d0dea0d Store Exception RBD incomplete write (Wrote only 8388608 out of 8392630 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8392630 bytes) Mar 11 19:17:36 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32128]: 2021-03-11 19:17:36.434 32128 ERROR glance.api.v2.image_data [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Failed to upload image data due to internal error: rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8392630 bytes) Mar 11 19:17:36 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32128]: 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] Caught error: RBD incomplete write (Wrote only 8388608 out of 8392630 bytes): rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8392630 bytes) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi raise value 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi self.force_reraise() 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi raise value 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 630, in add 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi raise exc 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 586, in add 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi image.write(chunk, offset) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "rbd.pyx", line 3304, in rbd.requires_not_closed.wrapper 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi File "rbd.pyx", line 4393, in rbd.Image.write 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8392630 bytes) 2021-03-11 19:17:36.500 32128 ERROR glance.common.wsgi Mar 11 19:17:38 cloud-c1-1030989-glance-container-071516d6 glance-wsgi-api[32134]: 2021-03-11 19:17:38.214 32134 WARNING keystonemiddleware.auth_token [req-400c49d7-2e76-49ce-b344-08908ae1d764 2e99c29bb33b45c7a001b1e21ec5e279 9d9e18f6ee4d4bb1bf7491a9c1bcad92 - default default] A valid token was submitted as a service token, but it was not a valid service token. This is incorrect but backwards compatible behaviour. This will be removed in future releases.