Can not delete glance image with message permission denied after vips failover

[Tatyanka]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "219"
  build_id: "2015-03-23_19-51-26"
  nailgun_sha: "799cb09d8f6584283f2d2bc55dc50925e23fb4b2"
  python-fuelclient_sha: "3624051242c83fdbdd1df9a0e466797c06b75043"
  astute_sha: "631f96d5a09cc48bfbddcbf056b946c8a80438f0"
  fuellib_sha: "e06e2f08f8bca0227b6af26dd61e22db9666bcf0"
  ostf_sha: "a4cf5f218c6aea98105b10c97a4aed8115c15867"
  fuelmain_sha: "290be05c5aed834b6dcdb7d14660b0be6be52d64"

Steps to Reproduce:
os: Ubuntu
net provider: nova flat
1. Deploy ha cluster with 3 controllers and 2 computes
2. Run ostf
3. after successful passed tests, create snapshot
4. Revert environment
5. Sync time
6. Delete public vip
7. Delete management vip
8. When vips recovered, run ostf

Actual result:
Deleting image over glance v1 failed with next error

http://paste.openstack.org/show/195896/

[Stanislaw Bogatkin (sbogatkin)]

Tatyanka, could you attach a diagnostic snapshot?

[Bogdan Dobrelya (bogdando)]

How exactly do you delete VIPs?

[Tatyanka (tatyana-leontovich)]

Bogdan thank you for updating summary, I meant delete image

[Tatyanka (tatyana-leontovich)]

Also you can look at the http://jenkins-product.srt.mirantis.net:8080/job/6.1.system_test.ubuntu.thread_5/75/ and revert the environment

[Ivan Berezovskiy (iberezovskiy)]

Me and Sergey Kolekonov have started to research this issue.

[Sergey Kolekonov (skolekonov)]

Access rights on the /var/lib/glance/image-cache/{incomplete invalid queue} are incorrect right after deployment:
drwxr-xr-x 2 root root 6 Apr 19 17:35 incomplete
drwxr-xr-x 2 root root 6 Apr 9 10:36 invalid
drwxr-xr-x 2 root root 6 Apr 9 10:36 queue

That's why the trace "IOError: [Errno 13] Permission denied" is observed. It's not fatal and appears every time when an instances is booted from an image first time (a non-cached image).

We have also found following similar bugs:
https://bugs.launchpad.net/fuel/+bug/1443913
https://bugs.launchpad.net/fuel/+bug/1436352

[Ivan Berezovskiy (iberezovskiy)]

The main problem of missconfiguring access right to /var/lib/glance/image-cache/{incomplete,invalid,queue} directories in installation sequence of glance packages. First, we are installing glance registry package which depends on glance-common. Glance common package creates user, group, etc.. and executes:
find /var/lib/glance -xdev -type d -exec chown glance:glance {} \;

This command set correct right to glance directories. But! these /var/lib/glance/image-cache/{incomplete,invalid,queue} directories will be created only when we will install glance-api package. Puppet installs glance-registry package before glance-api. That's why this problem happens.

One of the solutions is to move code for creating /var/lib/glance/image-cache/{incomplete,invalid,queue} directories to glance-common package.

[OSCI Robot (oscirobot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/6160

[OSCI Robot (oscirobot)]

Reviewed: https://review.fuel-infra.org/6160
Committed: https://review.fuel-infra.org/gitweb?p=openstack-build/glance-build.git;a=commitdiff;h=91d9e1d25ba850b9862115a120183198d5948ae7
Submitter: Igor Yozhikov
Branch: openstack-ci/fuel-6.1/2014.2

commit 91d9e1d25ba850b9862115a120183198d5948ae7
Author: Ivan Berezovskiy <email address hidden>

Create glance directories in glance-common package

  We should create and setup access rights for directories
  in glance-common package to avoid possible issue with
  missconfiguring directories. We can install glance-common
  before glance-api and then directories
  /var/lib/glance/image-cache/{incomplete,invalid,queue}
  will be owned by "root" instead of "glance" user

  Related-bug: # 1435804

Change-Id: Iafa01fc17bdd363f88b8f68e8b3a61426cb8fe71

[Tatyanka (tatyana-leontovich)]

verified 525 iso