tripleo

Bug #1731032
Comment #29

Comment 29 for bug 1731032

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-29: Fix merged to heat (stable/queens)

#29

Reviewed: https://review.openstack.org/557311
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8819c6380f242d5fd28ef85bfc63ad53f074aaad
Submitter: Zuul
Branch: stable/queens

commit 8819c6380f242d5fd28ef85bfc63ad53f074aaad
Author: Zane Bitter <email address hidden>
Date: Fri Nov 17 16:15:05 2017 -0500

Resolve race in providing deployment data to Swift

    While we are careful to ensure there are no race conditions in updating
    the software deployment metadata in the Heat database, there was no such
    protection in place for the actual update of the metadata provided to
    the server via Swift. In legacy Heat, this race could only occur when
    software deployments were created from the REST API or from separate
    stacks deploying software to the same server, because resources in the
    same stack were handled by a single thread making synchronous calls to
    create the software deployments. However, with convergence, resources in
    the same stack not only can be processed concurrently but will tend to
    be synchronised if they have the same dependencies (since they'll be
    triggered by consecutive asynchronous RPC messages).

This handles the race by retrying if a concurrent update is detected by
the time the data has been provided to Swift.

    Note that this is only guaranteed to result in the latest data being
    available on the assumption that a later write to Swift cannot overwrite
    an earlier one that has been accepted. This is probably *not* guaranteed
    by Swift in the presence of clock skew. Nonetheless, this should
    drastically reduce the failure rate.

    Similar caveats would probably apply to Zaqar, although Zaqar nominally
    provides FIFO queues (in practice, it likely depends on the backend
    used). However, with Zaqar clients receive all updates, not just the
    latest one, and os-collect-config stores the deployments contained in
    every message. Thus, Zaqar is not affected even assuming out-of-order
    arrival.

    Change-Id: Ic9a447f27e8c51f91f47f93b0fd1e9710341ec38
    Closes-Bug: #1731032
    (cherry picked from commit 46132999c016ed44972b9ac87a53946e9d8a7111)

Reviewed:  https://review.openstack.org/557311
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8819c6380f242d5fd28ef85bfc63ad53f074aaad
Submitter: Zuul
Branch:    stable/queens

commit 8819c6380f242d5fd28ef85bfc63ad53f074aaad
Author: Zane Bitter <zbitter@redhat.com>
Date:   Fri Nov 17 16:15:05 2017 -0500

Resolve race in providing deployment data to Swift
    
    While we are careful to ensure there are no race conditions in updating
    the software deployment metadata in the Heat database, there was no such
    protection in place for the actual update of the metadata provided to
    the server via Swift. In legacy Heat, this race could only occur when
    software deployments were created from the REST API or from separate
    stacks deploying software to the same server, because resources in the
    same stack were handled by a single thread making synchronous calls to
    create the software deployments. However, with convergence, resources in
    the same stack not only can be processed concurrently but will tend to
    be synchronised if they have the same dependencies (since they'll be
    triggered by consecutive asynchronous RPC messages).
    
    This handles the race by retrying if a concurrent update is detected by
    the time the data has been provided to Swift.
    
    Note that this is only guaranteed to result in the latest data being
    available on the assumption that a later write to Swift cannot overwrite
    an earlier one that has been accepted. This is probably *not* guaranteed
    by Swift in the presence of clock skew. Nonetheless, this should
    drastically reduce the failure rate.
    
    Similar caveats would probably apply to Zaqar, although Zaqar nominally
    provides FIFO queues (in practice, it likely depends on the backend
    used). However, with Zaqar clients receive all updates, not just the
    latest one, and os-collect-config stores the deployments contained in
    every message. Thus, Zaqar is not affected even assuming out-of-order
    arrival.
    
    Change-Id: Ic9a447f27e8c51f91f47f93b0fd1e9710341ec38
    Closes-Bug: #1731032
    (cherry picked from commit 46132999c016ed44972b9ac87a53946e9d8a7111)