[9.1] update from 9.0 to 9.1 is failed due to incorrect permissions for swift

[Denis Meltsaykin]

Update from 9.0GA to 9.1 has failed, the following errors are observed:

puppet-apply.log:
2016-08-26T15:32:05.120035+00:00 err: (/Stage[main]/Swift::Dispersion/Exec[swift-dispersion-populate]/returns) change from notrun to 0 failed: swift-dispersion-populate returned 1 instead of one of [0]

More info: https://paste.mirantis.net/show/2570/

swift-account-server.log:
2016-08-30T14:54:59.996611+00:00 err: ERROR __call__ error with PUT /2/373/AUTH_0f458b2c566442f7aa2c4541e5d4f8c9 : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 272, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1412, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 142, in PUT#012 broker.initialize(timestamp.internal)#012 File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 239, in initialize#012 mkdirs(self.db_dir)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1037, in mkdirs#012 os.makedirs(path)#012 File "/usr/lib/python2.7/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.7/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.7/os.py", line 157, in makedirs#012 mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/var/lib/glance/node/2/accounts/373' (txn: tx845ae69c819245e7b30d9-0057c59e43)

It looks like swifts' folder has wrong permissions inside:

root@node-1:~# ll /var/lib/glance/node/1/
total 0
drwxr-xr-x 7 swift swift 82 Aug 26 13:59 ./
drwxr-xr-x 4 swift swift 22 Aug 26 13:59 ../
drwxr-xr-x 3 glance glance 16 Aug 26 13:59 accounts/
drwxr-xr-x 2 glance glance 6 Aug 26 14:08 async_pending/
drwxr-xr-x 10 glance glance 85 Aug 26 14:08 containers/
drwxr-xr-x 6 glance glance 45 Aug 26 14:41 objects/
drwxr-xr-x 2 glance glance 6 Aug 26 14:08 tmp/

PS. Cannot generate a diagnostic snapshot due to: the "Timmy exit code: 127" issue.

[Ivan Berezovskiy (iberezovskiy)]

Problem is next:
if we update glance-common package it update ownership of /var/lib/glance directory.

Before update it was:

root@node-3:~# ll /var/lib/glance/
total 4
drwxr-x--- 5 glance swift 48 Aug 29 12:59 ./
drwxr-xr-x. 68 root root 4096 Aug 29 13:01 ../
drwxr-xr-x 5 glance glance 64 Aug 29 13:00 image-cache/
drwxr-xr-x 2 glance glance 6 Aug 22 12:48 images/
drwxr-xr-x 4 swift swift 22 Aug 29 12:59 node/

After:

drwxr-x--- 5 glance glance 48 Aug 25 15:29 ./
drwxr-xr-x. 68 root root 4096 Aug 25 15:37 ../
drwxr-xr-x 5 glance glance 64 Aug 25 15:30 image-cache/
drwxr-xr-x 2 glance glance 6 Jun 6 08:07 images/
drwxr-xr-x 4 glance glance 22 Aug 25 15:29 node/

So, swift user is unable to write to /var/lib/glance/node directory. It's truly Fuel issue, because we use /var/lib/glance/ as glance and swift storage in the same time. That's why it should fixed on Fuel side: we should add swift user to glance group and set 0775 mode for /var/lib/glance/ recursively

[Ivan Berezovskiy (iberezovskiy)]

It would be better to restore previous permissions right after the upgrade:
chown glance:swift /var/lib/glance/
chown -R swift:swift /var/lib/glance/node

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/364288

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/363820
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1e520c05ee221955adcf545b5a43b046d2e80233
Submitter: Jenkins
Branch: master

commit 1e520c05ee221955adcf545b5a43b046d2e80233
Author: iberezovskiy <email address hidden>
Date: Wed Aug 31 20:10:19 2016 +0300

    Restore swift storage directory permissions after upgrade

    Any update or reinstall glance-common package triggers
    its postinst action with following command:
    chown glance:glance -R /var/lib/glance/ /etc/glance/
    We use /var/lib/glance/node by default as swift storage
    folder http://goo.gl/97VJG2, and reinstallation of
    glance-common breaks swift user access to it, so swift
    services stop working. So, we need to fix /var/lib/glance/node
    folder permissions right after upgrade command.
    Also, the case of adding new controller node for already
    updated cluster is covered.
    This workaround is applied on when swift is enabled.

    Closes-bug: #1618553
    Related-bug: #1619282

    Change-Id: Iea6ce51d6ed9fe42bba1b26b58ba43268a7f880c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/364288
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a02f3892a93705020b3f18b650f44bbd376b115f
Submitter: Jenkins
Branch: stable/mitaka

commit a02f3892a93705020b3f18b650f44bbd376b115f
Author: iberezovskiy <email address hidden>
Date: Wed Aug 31 20:10:19 2016 +0300

    Restore swift storage directory permissions after upgrade

    Any update or reinstall glance-common package triggers
    its postinst action with following command:
    chown glance:glance -R /var/lib/glance/ /etc/glance/
    We use /var/lib/glance/node by default as swift storage
    folder http://goo.gl/97VJG2, and reinstallation of
    glance-common breaks swift user access to it, so swift
    services stop working. So, we need to fix /var/lib/glance/node
    folder permissions right after upgrade command.
    Also, the case of adding new controller node for already
    updated cluster is covered.
    This workaround is applied on when swift is enabled.

    Closes-bug: #1618553
    Related-bug: #1619282

    Change-Id: Iea6ce51d6ed9fe42bba1b26b58ba43268a7f880c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.