We have a deployment with three controller nodes running glance version 2:12.0.0-0ubuntu2 with a Ceph backend. About 50% of the image download requests from compute nodes are failing with an error like this:
2016-09-20 07:37:32.014 29606 WARNING glance.location [req-78e5191c-e31b-428e-82c5-30ef87feef4e a73f3a58c6ad478a838c3ca9353704f7 7a17e2b80f274591aec96baec3f6bf68 - - -] Get image 41a6aadf-c655-49c8-b77d-eedd356f5349 data failed: 'utf8' codec can't decode byte 0x82 in position 24: invalid start byte.
Looking at the code, it looks like the error is occuring in glance_store/_drivers/rbd.py while reading the image, but it seems very strange that glance is trying to utf8-decode a raw image, which pretty likely is bound to fail. But then it is unclear why only every other request is failing.
The error can be reproduced by just doing a "glance image-download", so it's nothing special happening from the Nova side.
We have a deployment with three controller nodes running glance version 2:12.0.0-0ubuntu2 with a Ceph backend. About 50% of the image download requests from compute nodes are failing with an error like this:
2016-09-20 07:37:32.014 29606 WARNING glance.location [req-78e5191c- e31b-428e- 82c5-30ef87feef 4e a73f3a58c6ad478 a838c3ca9353704 f7 7a17e2b80f27459 1aec96baec3f6bf 68 - - -] Get image 41a6aadf- c655-49c8- b77d-eedd356f53 49 data failed: 'utf8' codec can't decode byte 0x82 in position 24: invalid start byte.
Looking at the code, it looks like the error is occuring in glance_ store/_ drivers/ rbd.py while reading the image, but it seems very strange that glance is trying to utf8-decode a raw image, which pretty likely is bound to fail. But then it is unclear why only every other request is failing.
The error can be reproduced by just doing a "glance image-download", so it's nothing special happening from the Nova side.