When do heat stress tess ,many heat stack failed to delete due to "resources.volume_attachment: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400)"
the related logs are as follows:
The log trace I found:
heat-engine.log
2016-01-18 17:38:20.913 14651 DEBUG cinderclient.client [-] Unauthorized, reauthenticating. _cs_request /usr/lib/python2.7/dist-packages/cinderclient/client.py:277
2016-01-18 17:38:20.921 14651 INFO heat.engine.resource [-] DELETE: CinderVolume "volume" [1d68b786-0d15-40ad-b7f6-225ba7cb5e5b] Stack "s_rally_105da157_cGP4KXvB-wordpress_cluster-7c5sb4ybwz7x-3-3ix6jspitzq4-wordpress_volumes-wexhadnwdidf-0-ygpmrtsto6au" [17ebe95c-e230-4654-851c-dd6cc7c8f445]
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource Traceback (most recent call last):
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 500, in _action_recorder
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource yield
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 943, in delete
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource yield self.action_handler_task(action, *action_args)
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 313, in wrapper
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource step = next(subtask)
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 544, in action_handler_task
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource while not check(handler_data):
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/aws/ec2/volume.py", line 202, in check_delete_complete
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource return delete_task.step()
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 223, in step
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource next(self._runner)
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 324, in wrapper
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource subtask = next(parent)
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/aws/ec2/volume.py", line 186, in _delete
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource self.client_plugin().ignore_not_found(ex)
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/clients/client_plugin.py", line 171, in ignore_not_found
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource raise ex
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400)
2016-01-18 17:38:20.921 14651 TRACE heat.engine.resource
cinder-api.log
cinder-api.log.1:2016-01-18 17:38:20.912 13794 INFO eventlet.wsgi.server [-] 10.155.21.157,10.155.21.162 - - [18/Jan/2016 17:38:20] "GET /v2/9decb24445704034b4daa94206bae1aa/volumes/1d68b786-0d15-40ad-b7f6-225ba7cb5e5b HTTP/1.1" 401 281 0.001667
keystone.log
keystone.log.1:2016-01-18 17:38:20.917 20325 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:229
keystone.log.1:2016-01-18 17:38:20.919 20325 INFO keystone.common.wsgi [-] POST /tokens?
keystone.log.1:2016-01-18 17:38:20.919 20325 WARNING keystone.common.wsgi [-] Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.
keystone.log.1:2016-01-18 17:38:20.920 20325 INFO eventlet.wsgi.server [-] 10.155.21.157,10.155.21.162 - - [18/Jan/2016 17:38:20] "POST /v2.0/tokens HTTP/1.1" 400 441 0.003201
seems that the token is expired but due to heat did not pass the username and password to cinder client when initializing, cinder can not renew the token and reauthenticate.
You can use reauthenticatio