| 2016-09-20 07:50:28 |
Dr. Jens Harbott |
bug |
|
|
added bug |
| 2016-09-20 09:03:17 |
Hendrik Frese |
bug |
|
|
added subscriber Hendrik Frese |
| 2016-09-20 10:04:55 |
Dr. Jens Harbott |
bug task added |
|
ceph (Ubuntu) |
|
| 2016-09-20 10:07:36 |
Dr. Jens Harbott |
summary |
Glance fails to retrieve image from rbd backend |
image.stat() call sometimes fails |
|
| 2016-09-20 10:07:36 |
Dr. Jens Harbott |
description |
We have a deployment with three controller nodes running glance version 2:12.0.0-0ubuntu2 with a Ceph backend. About 50% of the image download requests from compute nodes are failing with an error like this:
2016-09-20 07:37:32.014 29606 WARNING glance.location [req-78e5191c-e31b-428e-82c5-30ef87feef4e a73f3a58c6ad478a838c3ca9353704f7 7a17e2b80f274591aec96baec3f6bf68 - - -] Get image 41a6aadf-c655-49c8-b77d-eedd356f5349 data failed: 'utf8' codec can't decode byte 0x82 in position 24: invalid start byte.
Looking at the code, it looks like the error is occuring in glance_store/_drivers/rbd.py while reading the image, but it seems very strange that glance is trying to utf8-decode a raw image, which pretty likely is bound to fail. But then it is unclear why only every other request is failing.
The error can be reproduced by just doing a "glance image-download", so it's nothing special happening from the Nova side. |
The error looks like
'utf8' codec can't decode byte 0x82 in position 24: invalid start byte.
where the position always is 24, which seems to be the end of the "block_name_prefix" string, and the byte itself it varying.
About 50% of the time the test script gives an error. |
|
| 2016-09-20 10:07:36 |
Dr. Jens Harbott |
tags |
rbd |
|
|
| 2016-09-20 10:34:16 |
Launchpad Janitor |
ceph (Ubuntu): status |
New |
Confirmed |
|
| 2016-09-20 10:34:16 |
Launchpad Janitor |
glance (Ubuntu): status |
New |
Confirmed |
|
| 2016-09-20 10:46:45 |
Dr. Jens Harbott |
bug task deleted |
glance (Ubuntu) |
|
|
| 2016-11-15 14:01:51 |
James Page |
ceph (Ubuntu): status |
Confirmed |
Triaged |
|
| 2016-11-15 14:01:53 |
James Page |
ceph (Ubuntu): importance |
Undecided |
High |
|
| 2016-11-15 14:01:59 |
James Page |
nominated for series |
|
Ubuntu Yakkety |
|
| 2016-11-15 14:01:59 |
James Page |
bug task added |
|
ceph (Ubuntu Yakkety) |
|
| 2016-11-15 14:01:59 |
James Page |
nominated for series |
|
Ubuntu Zesty |
|
| 2016-11-15 14:01:59 |
James Page |
bug task added |
|
ceph (Ubuntu Zesty) |
|
| 2016-11-15 14:01:59 |
James Page |
nominated for series |
|
Ubuntu Xenial |
|
| 2016-11-15 14:01:59 |
James Page |
bug task added |
|
ceph (Ubuntu Xenial) |
|
| 2016-11-15 14:02:06 |
James Page |
ceph (Ubuntu Yakkety): status |
New |
Triaged |
|
| 2016-11-15 14:02:08 |
James Page |
ceph (Ubuntu Xenial): status |
New |
Triaged |
|
| 2016-11-15 14:02:11 |
James Page |
ceph (Ubuntu Yakkety): importance |
Undecided |
High |
|
| 2016-11-15 14:02:12 |
James Page |
ceph (Ubuntu Xenial): importance |
Undecided |
High |
|
| 2017-01-06 10:19:30 |
James Page |
ceph (Ubuntu Zesty): status |
Triaged |
Fix Committed |
|
| 2017-01-18 10:09:18 |
James Page |
ceph (Ubuntu Zesty): status |
Fix Committed |
Fix Released |
|
