IOError: [Errno 13] Permission denied: '/etc/swift/container.builder'

[Anastasia Palkina]

"build_id": "2014-12-18_01-32-01",
"ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
"build_number": "56",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
"production": "docker",
"fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9",
"astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
"feature_groups": ["mirantis"],
"release": "6.0",
"release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}},
"fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"

1. Create new environment (CentOS, HA mode)
2. Choose neutron, GRE
3. Choose Ceilometer
4. Add 3 controller+cinder+mongo, 1 compute, 2 cinder+mongo, 2 mongo
5. Start deployment. It has failed
6. Primary controller (node-51) has gone away. I reset this node
7. There are many warnings, alerts and emerg errors in haproxy.log on primary controller (node-51):

http://paste.openstack.org/show/154782/

[Anastasia Palkina (apalkina)]

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaRU1yOFUtV2RhUEU/view?usp=sharing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/144122

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/144127

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144122
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b8a2310e9db883dc6e6d1e23a3afaad30f0ca922
Submitter: Jenkins
Branch: master

commit b8a2310e9db883dc6e6d1e23a3afaad30f0ca922
Author: Aleksandr Didenko <email address hidden>
Date: Fri Dec 26 13:22:03 2014 +0200

    Create ring builders under swift user

    Since we're running ring rebalance unser 'swift' user, we also
    need to create ring builders under the same user.

    Change-Id: I19a90cf0eea8718213fa7438618dc479a7b0ac3d
    Closes-bug: #1405755

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144127
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=199d6df0090b9ea53695c9d1b379c7776ccb2482
Submitter: Jenkins
Branch: stable/6.0

commit 199d6df0090b9ea53695c9d1b379c7776ccb2482
Author: Aleksandr Didenko <email address hidden>
Date: Fri Dec 26 13:22:03 2014 +0200

    Create ring builders under swift user

    Since we're running ring rebalance unser 'swift' user, we also
    need to create ring builders under the same user.

    Change-Id: I19a90cf0eea8718213fa7438618dc479a7b0ac3d
    Closes-bug: #1405755

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/144142

[Artem Panchenko (apanchenko-8)]

verified fix on 6.0 iso:

http://paste.openstack.org/show/154862/

[Anastasia Palkina (apalkina)]

Verified on ISO #76

"build_id": "2015-01-23_22-54-45", "ostf_sha": "95054878eff472eb6a0d9d185ce65e3105dde632", "build_number": "76", "auth_required": true, "api": "1.0", "nailgun_sha": "e75ac13a46d6c40fe7c90af51c053c0e0a818808", "production": "docker", "fuelmain_sha": "", "astute_sha": "48829a27b8041c36b14e34cc2ae760c8f57703a9", "feature_groups": ["mirantis"], "release": "6.1", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-01-23_22-54-45", "ostf_sha": "95054878eff472eb6a0d9d185ce65e3105dde632", "build_number": "76", "api": "1.0", "nailgun_sha": "e75ac13a46d6c40fe7c90af51c053c0e0a818808", "production": "docker", "fuelmain_sha": "", "astute_sha": "48829a27b8041c36b14e34cc2ae760c8f57703a9", "feature_groups": ["mirantis"], "release": "6.1", "fuellib_sha": "4c61bf87a96453a750df3e81606ba9fe21c2059f"}}}, "fuellib_sha": "4c61bf87a96453a750df3e81606ba9fe21c2059f"

[root@fuel ~]# ssh node-12 "swift-ring-builder /etc/swift/object.builder"
Warning: Permanently added 'node-12' (RSA) to the list of known hosts.
/etc/swift/object.builder, build version 6
2048 partitions, 3.000000 replicas, 1 regions, 3 zones, 6 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
             0 1 12 192.168.1.4 6000 192.168.1.4 6000 2 1.00 1024 0.00
             1 1 12 192.168.1.4 6000 192.168.1.4 6000 1 1.00 1024 0.00
             2 1 15 192.168.1.7 6000 192.168.1.7 6000 1 1.00 1024 0.00
             3 1 16 192.168.1.8 6000 192.168.1.8 6000 1 1.00 1024 0.00
             4 1 16 192.168.1.8 6000 192.168.1.8 6000 2 1.00 1024 0.00
             5 1 15 192.168.1.7 6000 192.168.1.7 6000 2 1.00 1024 0.00

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandr Didenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/144142
Reason: Rebalance may return 1 exit code and it's OK.