heat-engine fails to start when db is unavailable

[Steve Baker]

When heat-engine is started and there is no database currently available there is an error raised by EngineService.create_periodic_tasks. However if execution gets past create_periodic_tasks() then systemd is notified that the service is up, so heat will recover when the database comes back.

create_periodic_tasks needs to create its watch tasks inside a thread and retry until success so that heat-engine can start in the absence of a database.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/280451

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/280455

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/280456

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280451
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d4e270cd9faffead520eea8bf03bdb74b4b1a160
Submitter: Jenkins
Branch: master

commit d4e270cd9faffead520eea8bf03bdb74b4b1a160
Author: Steve Baker <email address hidden>
Date: Mon Feb 15 19:06:52 2016 -0500

    Create watch tasks in its own thread

    This change moves the creation of stack watch tasks into its own thread
    which retries until it has been done without error.

    The intent is to prevent create_periodic_tasks from raising an error
    when the database is not available, and retry stack watch creation until
    the database becomes available.

    Once execution gets past create_periodic_tasks() then systemd is
    notified that the service is up, and heat will recover when the database
    comes back.

    Change-Id: I4d61022db505581f0c1b2feed009c56df18a9ee5
    Closes-Bug: #1545885

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dave Walker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/280455
Reason: Kilo is now approaching EOL. We are in freeze pending the final release and no freeze exception has been raised. Therefore I am abandoning this change, if it is required in the release - please restore and raise a request. Thanks

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280456
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=0375ecb78aa941cd21c5cf916066c7c8e58cc7ac
Submitter: Jenkins
Branch: stable/liberty

commit 0375ecb78aa941cd21c5cf916066c7c8e58cc7ac
Author: Steve Baker <email address hidden>
Date: Mon Feb 15 19:06:52 2016 -0500

    Create watch tasks in its own thread

    This change moves the creation of stack watch tasks into its own thread
    which retries until it has been done without error.

    The intent is to prevent create_periodic_tasks from raising an error
    when the database is not available, and retry stack watch creation until
    the database becomes available.

    Once execution gets past create_periodic_tasks() then systemd is
    notified that the service is up, and heat will recover when the database
    comes back.

    Change-Id: I4d61022db505581f0c1b2feed009c56df18a9ee5
    Closes-Bug: #1545885
    (cherry picked from commit d4e270cd9faffead520eea8bf03bdb74b4b1a160)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 5.0.2 release.