race condition where auditor may start before swift.conf

[Emilien Macchi]

Deploying Swift with this manifest: https://github.com/stackforge/puppet-openstack-cloud/blob/master/manifests/object/storage.pp

Result:
Look at the timestamps, the swift auditor is started before /etc/swift/swift.conf was changed and most likely used the default hash suffix.
The proxy was started later, thus using the new hash_suffix and storing the object in a different place than the auditor expects.

[root@swift-object]# grep auditor /var/log/swift/swift.log | head -n 5
Apr 9 16:18:32 os-ci-test10 object-auditor: Begin object audit "forever" mode (ZBF)
Apr 9 16:18:32 os-ci-test10 object-auditor: Begin object audit "forever" mode (ZBF)
Apr 9 16:18:32 os-ci-test10 object-auditor: Begin object audit "forever" mode (ALL)
Apr 9 16:18:32 os-ci-test10 object-auditor: Begin object audit "forever" mode (ALL)
Apr 9 16:18:32 os-ci-test10 object-auditor: Object audit (ZBF) "forever" mode completed: 0.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 0.00, Total bytes/sec: 0.00, Auditing time: 0.00, Rate: 0.00

[root@swift-object]# stat /etc/swift/swift.conf
  File: '/etc/swift/swift.conf'
  Size: 55 Blocks: 8 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 787453 Links: 1
Access: (0660/-rw-rw----) Uid: ( 160/ swift) Gid: ( 160/ swift)
Context: system_u:object_r:etc_t:s0
Access: 2015-04-09 16:18:36.429054332 +0000
Modify: 2015-04-09 16:18:35.640024769 +0000
Change: 2015-04-09 16:18:35.640024769 +0000
 Birth: -

[root@swift-object]# ps aux | grep swift-proxy
root 4745 0.0 0.0 112612 688 pts/0 S+ 18:43 0:00 grep --color=auto swift-proxy
swift 24786 0.0 0.1 278928 30776 ? Ss 16:42 0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 24811 0.0 0.1 282276 32508 ? S 16:42 0:02 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 24812 0.0 0.1 282420 32548 ? S 16:42 0:02 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

[Christian Schwede (cschwede)]

The impact of this is that all objects will be quarantined and are no longer available. It only happens directly after installation; if the service or node is restarted before storing objects you will never see this bug.

Note that this requires a lot of manual work for an operators, because you need to manually move the quarantined objects back to their original location.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/172183

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172183
Committed: https://git.openstack.org/cgit/stackforge/puppet-swift/commit/?id=6dc85b0f7918fa295974f7afbb1a6b8a26f8f953
Submitter: Jenkins
Branch: master

commit 6dc85b0f7918fa295974f7afbb1a6b8a26f8f953
Author: Emilien Macchi <email address hidden>
Date: Thu Apr 9 15:23:01 2015 -0400

    Notify services if swift.conf is modified

    swift.conf contains the hash_suffix. With the current module, we have a
    race condition where services can start *before* having the right hash
    in swift.conf, so some objects mays be put in quarantine.

    When usint swift_config (which affect swift.conf), we should restart all
    services to ensure our ring is running correctly.

    Change-Id: I1087ba4ec079ca9c43bea9ff0511347632871ec2
    Closes-bug: #1442329

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/172402

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172402
Committed: https://git.openstack.org/cgit/stackforge/puppet-swift/commit/?id=4539c84521729e083db6dce052b915e1f6c59ebe
Submitter: Jenkins
Branch: stable/juno

commit 4539c84521729e083db6dce052b915e1f6c59ebe
Author: Emilien Macchi <email address hidden>
Date: Thu Apr 9 15:23:01 2015 -0400

    Notify services if swift.conf is modified

    swift.conf contains the hash_suffix. With the current module, we have a
    race condition where services can start *before* having the right hash
    in swift.conf, so some objects mays be put in quarantine.

    When usint swift_config (which affect swift.conf), we should restart all
    services to ensure our ring is running correctly.

    Change-Id: I1087ba4ec079ca9c43bea9ff0511347632871ec2
    Closes-bug: #1442329
    (cherry picked from commit 6dc85b0f7918fa295974f7afbb1a6b8a26f8f953)