heat stack delete failure

Bug #1653674 reported by Rajeev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
New
Undecided
Unassigned

Bug Description

Rally scenario - create-and-delete-stack-with-volume is failing consistently with:

GetResourceErrorStatus: Resource <Stack {u'disable_rollback': True, u'description': u'No description', u'parent': None, u'tags': None, u'stack_name': u's_rally_8847caab_TDJiDF79', u'stack_user_project_id': u'a5fe3817ccbd412ca6248f81fe03fc64', u'stack_status_reason': u'Resource DELETE failed: ResourceInError: resources.server: Went to status ERROR due to "Server s_rally_8847caab_TDJiDF79-server-745wfawgvcll delete failed: (500) Failed to terminate process 29164 with SIGKILL: Device or resource busy"', u'creation_time': u'2017-01-03T02:55:12', u'links': [{u'href': u'http://10.0.0.24:8004/v1/0f1ef63156254f1ba84fcea525a68f10/stacks/s_rally_8847caab_TDJiDF79/180ff02b-91e9-49fe-820b-ad578872536e', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'parameters': {u'OS::project_id': u'0f1ef63156254f1ba84fcea525a68f10', u'OS::stack_id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'OS::stack_name': u's_rally_8847caab_TDJiDF79', u'availability_zone': u'nova', u'image': u'cirros', u'volume_size': u'1', u'flavor': u'm1.tiny'}, u'timeout_mins': None, u'stack_status': u'DELETE_FAILED', u'stack_owner': u'c_rally_8847caab_yGw8hNVo', u'updated_time': None, u'id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'template_description': u'No description'}> has DELETE_FAILED status.
Fault: Status in failure list set(['DELETE_FAILED'])

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rally/task/runner.py", line 67, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/heat/stacks.py", line 79, in create_and_delete_stack
    self._delete_stack(stack)
  File "/usr/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/heat/utils.py", line 225, in _delete_stack
    check_interval=CONF.benchmark.heat_stack_delete_poll_interval)
  File "/usr/lib/python2.7/site-packages/rally/task/utils.py", line 242, in wait_for_status
    fault="Status in failure list %s" % str(failure_statuses))
GetResourceErrorStatus: Resource <Stack {u'disable_rollback': True, u'description': u'No description', u'parent': None, u'tags': None, u'stack_name': u's_rally_8847caab_TDJiDF79', u'stack_user_project_id': u'a5fe3817ccbd412ca6248f81fe03fc64', u'stack_status_reason': u'Resource DELETE failed: ResourceInError: resources.server: Went to status ERROR due to "Server s_rally_8847caab_TDJiDF79-server-745wfawgvcll delete failed: (500) Failed to terminate process 29164 with SIGKILL: Device or resource busy"', u'creation_time': u'2017-01-03T02:55:12', u'links': [{u'href': u'http://10.0.0.24:8004/v1/0f1ef63156254f1ba84fcea525a68f10/stacks/s_rally_8847caab_TDJiDF79/180ff02b-91e9-49fe-820b-ad578872536e', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'parameters': {u'OS::project_id': u'0f1ef63156254f1ba84fcea525a68f10', u'OS::stack_id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'OS::stack_name': u's_rally_8847caab_TDJiDF79', u'availability_zone': u'nova', u'image': u'cirros', u'volume_size': u'1', u'flavor': u'm1.tiny'}, u'timeout_mins': None, u'stack_status': u'DELETE_FAILED', u'stack_owner': u'c_rally_8847caab_yGw8hNVo', u'updated_time': None, u'id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'template_description': u'No description'}> has DELETE_FAILED status.
Fault: Status in failure list set(['DELETE_FAILED'])

HOT parameters:
image - cirros
flavor - m1.tiny
volume size - 1GB

heat-engine.log
---------------
2017-01-03 08:25:10.428 9924 INFO heat.engine.service [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Creating stack s_rally_8847caab_TDJiDF79
2017-01-03 08:25:10.452 9924 INFO heat.engine.resource [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Validating Server "server"
2017-01-03 08:25:10.452 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:10.453 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:10.453 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:12.564 9924 INFO heat.engine.resource [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Validating CinderVolume "cinder_volume"
2017-01-03 08:25:12.565 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:12.565 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:12.565 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:12.566 9924 INFO heat.engine.resource [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Validating CinderVolumeAttachment "volume_attachment"
2017-01-03 08:25:12.567 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:12.567 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:12.567 9924 WARNING heat.engine.rsrc_defn [req-96c3a4f6-cab1-414a-9721-ac9b07ed4b49 c_rally_8847caab_yGw8hNVo c_rally_8847caab_zrro1OcC - default default] Reading the ResourceDefinition as if it were a snippet of a CloudFormation template is deprecated, and the ability to treat it as such will be removed in the future. Resource plugins should use the ResourceDefinition API to work with the definition of the resource instance.
2017-01-03 08:25:13.192 9924 INFO heat.engine.stack [-] Stack CREATE IN_PROGRESS (s_rally_8847caab_TDJiDF79): Stack CREATE started
2017-01-03 08:25:13.201 9924 INFO heat.engine.resource [-] creating Server "server" Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:25:14.288 9924 INFO heat.engine.resource [-] creating CinderVolume "cinder_volume" Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:25:14.305 9924 INFO heat.engine.clients.os.cinder [-] Creating Cinder client with volume API version 2.
2017-01-03 08:25:17.299 9924 INFO heat.engine.resource [-] creating CinderVolumeAttachment "volume_attachment" Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:25:18.627 9923 INFO heat.engine.service [req-c0610a3e-7160-442b-b49c-187172d4976d - - - - -] Service 2dc1a12a-e0b6-4f7d-ac16-6f29c0b7bbbe is updated
2017-01-03 08:25:18.631 9921 INFO heat.engine.service [req-334c4d2a-1ed6-4927-96c8-9ff480a6f4d1 - - - - -] Service aa688e36-fd59-4df9-9453-a025726fc957 is updated
2017-01-03 08:25:19.816 9924 INFO heat.engine.service [req-743d6797-8e68-4e8a-bd1c-110875feecbf - - - - -] Service 25e7a5df-5348-4eeb-a7b6-583de3307c37 is updated
2017-01-03 08:25:19.817 9922 INFO heat.engine.service [req-fc2a7c59-7e25-43a8-8974-7a99c3a41a5d - - - - -] Service 6ab04965-14a9-4a9f-b5a3-954bbcc989a5 is updated
2017-01-03 08:25:21.339 9924 INFO heat.engine.clients.os.cinder [-] Attaching volume 13811f64-42d3-42ef-8507-386dd01d2cf0 complete
2017-01-03 08:25:21.371 9924 INFO heat.engine.stack [-] Stack CREATE COMPLETE (s_rally_8847caab_TDJiDF79): Stack CREATE completed successfully
2017-01-03 08:25:21.977 9921 INFO heat.engine.service [req-eebe54b7-8a4b-434a-aabd-433ab67f48e3 - c_rally_8847caab_zrro1OcC - default default] Deleting stack s_rally_8847caab_TDJiDF79
2017-01-03 08:25:22.019 9921 INFO heat.engine.stack [-] Stack DELETE IN_PROGRESS (s_rally_8847caab_TDJiDF79): Stack DELETE started
2017-01-03 08:25:22.037 9921 INFO heat.engine.resource [-] deleting CinderVolumeAttachment "volume_attachment" [13811f64-42d3-42ef-8507-386dd01d2cf0] Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:25:22.653 9921 INFO heat.engine.clients.os.cinder [-] Creating Cinder client with volume API version 2.
2017-01-03 08:25:25.257 9921 INFO heat.engine.clients.os.nova [-] Volume 13811f64-42d3-42ef-8507-386dd01d2cf0 is detached from server 72f04bf3-51f0-4586-82b8-0ad90d271217
2017-01-03 08:25:25.354 9921 INFO heat.engine.resource [-] deleting Server "server" [72f04bf3-51f0-4586-82b8-0ad90d271217] Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:25:26.380 9921 INFO heat.engine.resource [-] deleting CinderVolume "cinder_volume" [13811f64-42d3-42ef-8507-386dd01d2cf0] Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:26:27.700 9923 INFO heat.engine.service [req-48a391aa-6396-4380-8705-89a42eb745d8 - - - - -] Service 2dc1a12a-e0b6-4f7d-ac16-6f29c0b7bbbe is updated
2017-01-03 08:26:27.705 9921 INFO heat.engine.service [req-27e20a98-eb58-4d01-951d-e4a5faaee813 - - - - -] Service aa688e36-fd59-4df9-9453-a025726fc957 is updated
2017-01-03 08:26:27.706 9924 INFO heat.engine.service [req-9a426999-5263-48f5-bb0a-ab32879a5682 - - - - -] Service 25e7a5df-5348-4eeb-a7b6-583de3307c37 is updated
2017-01-03 08:26:27.708 9922 INFO heat.engine.service [req-1b4548dd-93ee-4521-987d-99ec5fe3c0bd - - - - -] Service 6ab04965-14a9-4a9f-b5a3-954bbcc989a5 is updated
2017-01-03 08:27:13.374 9921 INFO heat.engine.resource [-] DELETE: Server "server" [72f04bf3-51f0-4586-82b8-0ad90d271217] Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource Traceback (most recent call last):
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 704, in _action_recorder
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource yield
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1479, in delete
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource yield self.action_handler_task(action, *action_args)
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 314, in wrapper
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource step = next(subtask)
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 749, in action_handler_task
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource while not check(handler_data):
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 1502, in check_delete_complete
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource prg.server_id)
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 437, in check_delete_server_complete
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource status_reason=errmsg)
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource ResourceInError: Went to status ERROR due to "Server s_rally_8847caab_TDJiDF79-server-745wfawgvcll delete failed: (500) Failed to terminate process 29164 with SIGKILL: Device or resource busy"
2017-01-03 08:27:13.374 9921 ERROR heat.engine.resource
2017-01-03 08:27:13.878 9921 INFO heat.engine.stack [-] Stack DELETE FAILED (s_rally_8847caab_TDJiDF79): Resource DELETE failed: ResourceInError: resources.server: Went to status ERROR due to "Server s_rally_8847caab_TDJiDF79-server-745wfawgvcll delete failed: (500) Failed to terminate process 29164 with SIGKILL: Device or resource busy"
2017-01-03 08:27:15.134 9924 INFO heat.engine.service [req-1006409e-a285-45e2-b288-e9647b068777 - c_rally_8847caab_zrro1OcC - default default] Deleting stack s_rally_8847caab_TDJiDF79
2017-01-03 08:27:15.182 9924 INFO heat.engine.stack [-] Stack DELETE IN_PROGRESS (s_rally_8847caab_TDJiDF79): Stack DELETE started
2017-01-03 08:27:15.201 9924 INFO heat.engine.resource [-] deleting Server "server" [72f04bf3-51f0-4586-82b8-0ad90d271217] Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:27:15.586 9924 INFO heat.engine.resource [-] deleting CinderVolume "cinder_volume" [13811f64-42d3-42ef-8507-386dd01d2cf0] Stack "s_rally_8847caab_TDJiDF79" [180ff02b-91e9-49fe-820b-ad578872536e]
2017-01-03 08:27:16.706 9924 INFO heat.engine.clients.os.cinder [-] Creating Cinder client with volume API version 2.
2017-01-03 08:27:18.243 9924 INFO heat.engine.stack [-] Stack DELETE COMPLETE (s_rally_8847caab_TDJiDF79): Stack DELETE completed successfully
2017-01-03 08:27:18.621 9921 INFO heat.engine.service [req-143729b8-7151-44ba-bd3a-eb3f79565746 - - - - -] Service aa688e36-fd59-4df9-9453-a025726fc957 is updated
2017-01-03 08:27:18.622 9923 INFO heat.engine.service [req-b83de585-b2be-4a08-abd2-319a256a188d - - - - -] Service 2dc1a12a-e0b6-4f7d-ac16-6f29c0b7bbbe is updated
2017-01-03 08:27:19.820 9922 INFO heat.engine.service [req-9aa84c39-58c6-4302-be88-26f263aecb85 - - - - -] Service 6ab04965-14a9-4a9f-b5a3-954bbcc989a5 is updated
2017-01-03 08:27:19.821 9924 INFO heat.engine.service [req-57531ef5-1d1b-4b53-a8b8-86345b767789 - - - - -] Service 25e7a5df-5348-4eeb-a7b6-583de3307c37 is updated
2017-01-03 08:28:18.614 9921 INFO heat.engine.service [req-38fff1d7-d122-41e9-a9de-5f3825f2254b - - - - -] Service aa688e36-fd59-4df9-9453-a025726fc957 is updated
2017-01-03 08:28:18.620 9923 INFO heat.engine.service [req-28dc6ca9-f3fd-4d90-ace7-a5af075d9044 - - - - -] Service 2dc1a12a-e0b6-4f7d-ac16-6f29c0b7bbbe is updated
2017-01-03 08:28:19.811 9924 INFO heat.engine.service [req-6f784101-7c02-4e7f-a65c-b96c879b492f - - - - -] Service 25e7a5df-5348-4eeb-a7b6-583de3307c37 is updated
2017-01-03 08:28:19.818 9922 INFO heat.engine.service [req-ae77cf9f-f31c-424e-9f3b-3a5ab3b6a087 - - - - -] Service 6ab04965-14a9-4a9f-b5a3-954bbcc989a5 is updated

on the openstack setup:
----------------------
[root@localhost ~(keystone_admin)]# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[root@localhost ~(keystone_admin)]# cinder list
+----+--------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+----+--------+------+------+-------------+----------+-------------+
+----+--------+------+------+-------------+----------+-------------+
[root@localhost ~(keystone_admin)]# heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+----+------------+--------------+---------------+--------------+
| id | stack_name | stack_status | creation_time | updated_time |
+----+------------+--------------+---------------+--------------+
+----+------------+--------------+---------------+--------------+

Rally console logs:
------------------
 rally task start create-and-delete-stack-with-volume.json
--------------------------------------------------------------------------------
 Preparing input task
--------------------------------------------------------------------------------

Input task is:
{
    "HeatStacks.create_and_delete_stack": [
        {
            "args": {
               "template_path": "/root/Stock_Heat_Templates/server-with-volume.yaml.template"
            },
            "runner": {
                "type": "constant",
                "times": 1,
                "concurrency": 1
            },
            "context": {
                "users": {
                    "tenants": 1,
                    "users_per_tenant": 1
                }
            }
        }
    ]
}

Task syntax is correct :)
2017-01-03 08:25:05.895 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Task validation.
2017-01-03 08:25:05.912 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Task validation of scenarios names.
2017-01-03 08:25:05.914 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Task validation of scenarios names.
2017-01-03 08:25:05.915 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Task validation of syntax.
2017-01-03 08:25:05.919 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Task validation of syntax.
2017-01-03 08:25:05.919 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Task validation of semantic.
2017-01-03 08:25:05.919 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Task validation check cloud.
2017-01-03 08:25:06.179 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Task validation check cloud.
2017-01-03 08:25:06.186 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Enter context: `users`
2017-01-03 08:25:06.627 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Enter context: `users`
2017-01-03 08:25:07.015 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Exit context: `users`
2017-01-03 08:25:09.757 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Exit context: `users`
2017-01-03 08:25:09.757 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Task validation of semantic.
2017-01-03 08:25:09.757 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Task validation.
Task config is valid :)
--------------------------------------------------------------------------------
 Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad: started
--------------------------------------------------------------------------------

Benchmarking... This can take a while...

To track task status use:

        rally task status
        or
        rally task detailed

Using task: 8847caab-4e20-4d4f-9314-cdc23a3bd4ad
2017-01-03 08:25:09.766 17633 INFO rally.api [-] Benchmark Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad on Deployment b6c81337-8ce2-4df1-b743-79e2177df55e
2017-01-03 08:25:09.768 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Benchmarking.
2017-01-03 08:25:09.785 17633 INFO rally.task.engine [-] Running benchmark with key:
{
  "kw": {
    "runner": {
      "type": "constant",
      "concurrency": 1,
      "times": 1
    },
    "args": {
      "template_path": "/root/Stock_Heat_Templates/server-with-volume.yaml.template"
    },
    "context": {
      "users": {
        "users_per_tenant": 1,
        "tenants": 1
      }
    }
  },
  "name": "HeatStacks.create_and_delete_stack",
  "pos": 0
}
2017-01-03 08:25:09.790 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Enter context: `users`
2017-01-03 08:25:10.179 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Enter context: `users`
2017-01-03 08:25:10.193 17766 INFO rally.task.runner [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | ITER: 0 START
2017-01-03 08:25:15.315 17766 WARNING rally.common.logging [-] 'wait_for' is deprecated in Rally v0.1.2: Use wait_for_status instead.
2017-01-03 08:27:14.714 17766 INFO rally.task.runner [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | ITER: 0 END: Error GetResourceErrorStatus: Resource <Stack {u'disable_rollback': True, u'description': u'No description', u'parent': None, u'tags': None, u'stack_name': u's_rally_8847caab_TDJiDF79', u'stack_user_project_id': u'a5fe3817ccbd412ca6248f81fe03fc64', u'stack_status_reason': u'Resource DELETE failed: ResourceInError: resources.server: Went to status ERROR due to "Server s_rally_8847caab_TDJiDF79-server-745wfawgvcll delete failed: (500) Failed to terminate process 29164 with SIGKILL: Device or resource busy"', u'creation_time': u'2017-01-03T02:55:12', u'links': [{u'href': u'http://10.0.0.24:8004/v1/0f1ef63156254f1ba84fcea525a68f10/stacks/s_rally_8847caab_TDJiDF79/180ff02b-91e9-49fe-820b-ad578872536e', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'parameters': {u'OS::project_id': u'0f1ef63156254f1ba84fcea525a68f10', u'OS::stack_id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'OS::stack_name': u's_rally_8847caab_TDJiDF79', u'availability_zone': u'nova', u'image': u'cirros', u'volume_size': u'1', u'flavor': u'm1.tiny'}, u'timeout_mins': None, u'stack_status': u'DELETE_FAILED', u'stack_owner': u'c_rally_8847caab_yGw8hNVo', u'updated_time': None, u'id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'template_description': u'No description'}> has DELETE_FAILED status.
Fault: Status in failure list set(['DELETE_FAILED'])
2017-01-03 08:27:14.722 17633 INFO rally.plugins.openstack.context.cleanup.user [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: user resources cleanup
2017-01-03 08:27:19.468 17633 INFO rally.plugins.openstack.context.cleanup.user [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: user resources cleanup
2017-01-03 08:27:19.469 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Starting: Exit context: `users`
2017-01-03 08:27:21.060 17633 INFO rally.plugins.openstack.context.keystone.users [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Exit context: `users`
2017-01-03 08:27:22.188 17633 INFO rally.task.engine [-] Load duration is: 124.518434
2017-01-03 08:27:22.188 17633 INFO rally.task.engine [-] Full runner duration is: 124.5377
2017-01-03 08:27:22.189 17633 INFO rally.task.engine [-] Full duration is 131.271042
2017-01-03 08:27:22.277 17633 INFO rally.task.engine [-] Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad | Completed: Benchmarking.

--------------------------------------------------------------------------------
Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad: finished
--------------------------------------------------------------------------------

test scenario HeatStacks.create_and_delete_stack
args position 0
args values:
{
  "runner": {
    "type": "constant",
    "concurrency": 1,
    "times": 1
  },
  "args": {
    "template_path": "/root/Stock_Heat_Templates/server-with-volume.yaml.template"
  },
  "context": {
    "users": {
      "users_per_tenant": 1,
      "project_domain": "default",
      "user_choice_method": "random",
      "user_domain": "default",
      "tenants": 1,
      "resource_management_workers": 20
    }
  }
}

--------------------------------------------------------------------------------
 Task 8847caab-4e20-4d4f-9314-cdc23a3bd4ad has 1 error(s)
--------------------------------------------------------------------------------

GetResourceErrorStatus: Resource <Stack {u'disable_rollback': True, u'description': u'No description', u'parent': None, u'tags': None, u'stack_name': u's_rally_8847caab_TDJiDF79', u'stack_user_project_id': u'a5fe3817ccbd412ca6248f81fe03fc64', u'stack_status_reason': u'Resource DELETE failed: ResourceInError: resources.server: Went to status ERROR due to "Server s_rally_8847caab_TDJiDF79-server-745wfawgvcll delete failed: (500) Failed to terminate process 29164 with SIGKILL: Device or resource busy"', u'creation_time': u'2017-01-03T02:55:12', u'links': [{u'href': u'http://10.0.0.24:8004/v1/0f1ef63156254f1ba84fcea525a68f10/stacks/s_rally_8847caab_TDJiDF79/180ff02b-91e9-49fe-820b-ad578872536e', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'parameters': {u'OS::project_id': u'0f1ef63156254f1ba84fcea525a68f10', u'OS::stack_id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'OS::stack_name': u's_rally_8847caab_TDJiDF79', u'availability_zone': u'nova', u'image': u'cirros', u'volume_size': u'1', u'flavor': u'm1.tiny'}, u'timeout_mins': None, u'stack_status': u'DELETE_FAILED', u'stack_owner': u'c_rally_8847caab_yGw8hNVo', u'updated_time': None, u'id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'template_description': u'No description'}> has DELETE_FAILED status.
Fault: Status in failure list set(['DELETE_FAILED'])

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rally/task/runner.py", line 67, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/heat/stacks.py", line 79, in create_and_delete_stack
    self._delete_stack(stack)
  File "/usr/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/heat/utils.py", line 225, in _delete_stack
    check_interval=CONF.benchmark.heat_stack_delete_poll_interval)
  File "/usr/lib/python2.7/site-packages/rally/task/utils.py", line 242, in wait_for_status
    fault="Status in failure list %s" % str(failure_statuses))
GetResourceErrorStatus: Resource <Stack {u'disable_rollback': True, u'description': u'No description', u'parent': None, u'tags': None, u'stack_name': u's_rally_8847caab_TDJiDF79', u'stack_user_project_id': u'a5fe3817ccbd412ca6248f81fe03fc64', u'stack_status_reason': u'Resource DELETE failed: ResourceInError: resources.server: Went to status ERROR due to "Server s_rally_8847caab_TDJiDF79-server-745wfawgvcll delete failed: (500) Failed to terminate process 29164 with SIGKILL: Device or resource busy"', u'creation_time': u'2017-01-03T02:55:12', u'links': [{u'href': u'http://10.0.0.24:8004/v1/0f1ef63156254f1ba84fcea525a68f10/stacks/s_rally_8847caab_TDJiDF79/180ff02b-91e9-49fe-820b-ad578872536e', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'parameters': {u'OS::project_id': u'0f1ef63156254f1ba84fcea525a68f10', u'OS::stack_id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'OS::stack_name': u's_rally_8847caab_TDJiDF79', u'availability_zone': u'nova', u'image': u'cirros', u'volume_size': u'1', u'flavor': u'm1.tiny'}, u'timeout_mins': None, u'stack_status': u'DELETE_FAILED', u'stack_owner': u'c_rally_8847caab_yGw8hNVo', u'updated_time': None, u'id': u'180ff02b-91e9-49fe-820b-ad578872536e', u'template_description': u'No description'}> has DELETE_FAILED status.
Fault: Status in failure list set(['DELETE_FAILED'])

--------------------------------------------------------------------------------
+----------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+-------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+-------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| heat.delete_stack | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 1 |
| heat.create_stack | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 1 |
| total | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 1 |
+-------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 124.518434
Full duration: 131.271042

HINTS:
* To plot HTML graphics with this data, run:
        rally task report 8847caab-4e20-4d4f-9314-cdc23a3bd4ad --out output.html

* To generate a JUnit report, run:
        rally task report 8847caab-4e20-4d4f-9314-cdc23a3bd4ad --junit --out output.xml

* To get raw JSON output of task results, run:
        rally task results 8847caab-4e20-4d4f-9314-cdc23a3bd4ad

Revision history for this message
Rajeev (rsomayaj) wrote :

openstack version - 13.1.1

Changed in heat:
assignee: nobody → huangtianhua (huangtianhua)
Changed in heat:
assignee: huangtianhua (huangtianhua) → nobody
Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.