Stacks can time out before they've started with clock skew

[Zane Bitter]

The Stack.time_elapsed function only takes into account the seconds part of the difference between the current and start times, and ignores the days part. This can in theory cause some weird edge cases, but in practice the one in particular it will hit is that if the difference is small but negative then the delta is -1 days plus a large, positive number of seconds (*not* 0 days and a small negative number of seconds).

This can easily happen if clocks are out of sync, since we are comparing database time to local machine time. (Ideally we'd find a way to not do that.) In this case the stack will time out as soon as we try to check a resource on the server with the slow clock. Event logs may even appear to show that the timeout happened even before the action started.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/462602

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/462754

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462602
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f3f05c9a3f6109ec1e102ac9677accb82eb6cc54
Submitter: Jenkins
Branch: master

commit f3f05c9a3f6109ec1e102ac9677accb82eb6cc54
Author: Zane Bitter <email address hidden>
Date: Thu May 4 11:41:55 2017 -0400

    Deal correctly with negative elapsed time

    Since we're comparing local time to database time, we need to correctly
    handle negative offsets to account for clock skew. The datetime.timedelta
    class always reports a positive number of seconds, but with a possibly
    negative number of *days*. Therefore, take the days into account.

    Change-Id: I4422e4cf41cb57cdc89548d01b0abc27cda914ef
    Closes-Bug: #1688327

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/465727

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462754
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=159b73211e7669dd01ba483a8127c847fdc99deb
Submitter: Jenkins
Branch: master

commit 159b73211e7669dd01ba483a8127c847fdc99deb
Author: Zane Bitter <email address hidden>
Date: Thu May 4 17:24:59 2017 -0400

    Get rid of timeutils.round_to_seconds()

    This was introduced in 0bc4de65e3bd9dc9902e6833373c78df0badbcb4 to try
    to ensure that we never got a negative difference in times, which would
    trigger a bug in our handling of time differences.

    Now that bug 1688327 is fixed, there's no need for this complexity any
    longer.

    Change-Id: Iff76ff5ab2b1d642d464f111e78e248faf6ac77c
    Related-Bug: #1688327
    Related-Bug: #1493166

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/469937

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/465727
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c6586707736fc4b8a0061b8d3f529562c3f933f2
Submitter: Jenkins
Branch: stable/ocata

commit c6586707736fc4b8a0061b8d3f529562c3f933f2
Author: Zane Bitter <email address hidden>
Date: Thu May 4 11:41:55 2017 -0400

    Deal correctly with negative elapsed time

    Since we're comparing local time to database time, we need to correctly
    handle negative offsets to account for clock skew. The datetime.timedelta
    class always reports a positive number of seconds, but with a possibly
    negative number of *days*. Therefore, take the days into account.

    Change-Id: I4422e4cf41cb57cdc89548d01b0abc27cda914ef
    Closes-Bug: #1688327
    (cherry picked from commit f3f05c9a3f6109ec1e102ac9677accb82eb6cc54)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 9.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 8.0.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469937
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8c8e15c9ceb181a17b5c006fae2d48ad3a13e6fc
Submitter: Jenkins
Branch: stable/newton

commit 8c8e15c9ceb181a17b5c006fae2d48ad3a13e6fc
Author: Zane Bitter <email address hidden>
Date: Thu May 4 11:41:55 2017 -0400

    Deal correctly with negative elapsed time

    Since we're comparing local time to database time, we need to correctly
    handle negative offsets to account for clock skew. The datetime.timedelta
    class always reports a positive number of seconds, but with a possibly
    negative number of *days*. Therefore, take the days into account.

    Change-Id: I4422e4cf41cb57cdc89548d01b0abc27cda914ef
    Closes-Bug: #1688327
    (cherry picked from commit f3f05c9a3f6109ec1e102ac9677accb82eb6cc54)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/560331

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ethan Lynn (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/560331