Failed to boot VM due to Ceph I/O Error

[Alexey Yelistratov]

Performed light rally tests from 2015/09/16 14:09:26 to 2015/09/16 16:12:41.

VM instances failed to spawn due to ceph filesystem reading error.

All logs are here:
http://paste.openstack.org/show/467755/

HTTP GET on image via glance api returns 404 and it corresponds to -2 ENOENT error in python.log.

However, if I perform ls -l on ceph, the image is found and could be read.

Could you please help us find the reason of such strange behaviour?

root@node-3:~# rbd -p images ls -l | grep 6bb86ecf-75ac-47a9-8db9-f72034c9aa96
6bb86ecf-75ac-47a9-8db9-f72034c9aa96 12839k 2
6bb86ecf-75ac-47a9-8db9-f72034c9aa96@snap 12839k 2 yes

Cluster configuration:
baremetal, ubuntu trusty, IBP, HA, neutron-vlan, no DVR, Ceph-all, Nova-debug, Nova-quotas, 7.0-288

api: '1.0'
astute_sha: a717657232721a7fafc67ff5e1c696c9dbeb0b95
auth_required: true
build_id: '288'
build_number: '288'
feature_groups:
- mirantis
fuel-agent_sha: 082a47bf014002e515001be05f99040437281a2d
fuel-library_sha: 121016a09b0e889994118aa3ea42fa67eabb8f25
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 1f08e6e71021179b9881a824d9c999957fcc7045
fuelmain_sha: 6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85
nailgun_sha: 93477f9b42c5a5e0506248659f40bebc9ac23943
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 1ce8ecd8beb640f2f62f73435f4e18d1469979ac
release: '7.0

Diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-16_17-41-10.tar.xz

[Mike Fedosin (mfedosin)]

Hi! it's a known issue - glance was broken in iso 288. We've already fixed it (https://review.openstack.org/#/c/221409/ and https://review.openstack.org/#/c/221369/), so you have to take newer iso or change one config parameter, called 'use_user_token', in /etc/glance/glance-api.conf to True.

I hope it's the reason of this bug and we can set it status to Invalid.

[Alexey Yelistratov (ayelistratov-deactivatedaccount)]

Thanks a lot! I'll try this on iso 289+ and if everything works well, I'll close the issue.

[Vitaly Sedelnik (vsedelnik)]

Based on comment #1 setting this issue to Fix Committed

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 7.0. The issue is not reproduced.

ISO info:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "301"
  build_id: "301"
  nailgun_sha: "4162b0c15adb425b37608c787944d1983f543aa8"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0 kilo. The issue is not reproduced.

ISO info:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "126"
  build_id: "126"
  fuel-nailgun_sha: "70d8b7e80573728e04ac5478c112850afcfa9802"
  python-fuelclient_sha: "56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3"
  fuel-agent_sha: "e881f0dabd09af4be4f3e22768b02fe76278e20e"
  fuel-nailgun-agent_sha: "d66f188a1832a9c23b04884a14ef00fc5605ec6d"
  astute_sha: "0f753467a3f16e4d46e7e9f1979905fb178e4d5b"
  fuel-library_sha: "37734918592cef7d355f357d14cbf960540e09cd"
  fuel-ostf_sha: "41aa5059243cbb25d7a80b97f8e1060a502b99dd"
  fuelmain_sha: "51614465980e5f62a5796779d3f6c3305c1d5739"