Stack fails upon 503 response code from Nova API

[Jason Dunsmore]

When Nova responds with a 503, the stack fails:

2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource Traceback (most recent call last):
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/heat/engine/resource.py", line 422, in _do_action
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource while not check(handle_data):
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/heat/rackspace/rackspace/resources/cloud_server.py", line 408, in check_create_complete
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource if not self._check_active(server):
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/heat/engine/resources/server.py", line 390, in _check_active
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource nova_utils.refresh_server(server)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/heat/engine/resources/nova_utils.py", line 55, in refresh_server
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource server.get()
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/novaclient/openstack/common/apiclient/base.py", line 473, in get
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource new = self.manager.get(self.id)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 543, in get
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource return self._get("/servers/%s" % base.getid(server), "server")
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/novaclient/base.py", line 147, in _get
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource _resp, body = self.api.client.get(url)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/novaclient/client.py", line 254, in get
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource return self._cs_request(url, 'GET', **kwargs)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/novaclient/client.py", line 231, in _cs_request
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource **kwargs)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/novaclient/client.py", line 213, in _time_request
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource resp, body = self.request(url, method, **kwargs)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource File "/opt/heat/lib/python2.7/site-packages/novaclient/client.py", line 207, in request
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource raise exceptions.from_response(resp, body, url, method)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource ClientException: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 503)
2014-03-07 13:27:10.215 10952 TRACE heat.engine.resource

503 (Service Unavailable) is an intermittent, non-critical error and should be handled the same way OverLimit and 500 errors are being handled.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/79065

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79065
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2516c4413e95adfc2f56073688d60bb50218305d
Submitter: Jenkins
Branch: master

commit 2516c4413e95adfc2f56073688d60bb50218305d
Author: Jason Dunsmore <email address hidden>
Date: Fri Mar 7 13:38:46 2014 -0600

    Handle 503 response from Nova API

    The stack fails when Nova responds with a 503 during a
    create/update/delete. 503 (Service Unavailable) is an intermittent,
    non-critical error and should be handled the same way OverLimit and 500
    errors are being handled.

    Change-Id: Iadc3b39d099fcc7870764b7a42c5edee41c527dc
    Closes-Bug: #1289549