[root@fuel ~]# fuel --fuel-version
api: '1.0'
astute_sha: 1f87a9b9a47de7498b4061d15a8c7fb9435709d5
auth_required: true
build_id: 2015-02-13_16-48-42
build_number: '120'
feature_groups:
- mirantis
fuellib_sha: 7f8d4382abfcd4338964182ebfea1d539f963e66
fuelmain_sha: 4b6bcf60d712b74a3a53252bfbb5ca34aa7497c4
nailgun_sha: 1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666
ostf_sha: f9c37d0876141e1550eb4e703a8e500cd463282f
production: docker
python-fuelclient_sha: 61431ed16fc00039a269424bdbaa410277eff609
release: '6.1'
release_versions:
2014.2-6.1:
VERSION:
api: '1.0'
astute_sha: 1f87a9b9a47de7498b4061d15a8c7fb9435709d5
build_id: 2015-02-13_16-48-42
build_number: '120'
feature_groups:
- mirantis
fuellib_sha: 7f8d4382abfcd4338964182ebfea1d539f963e66
fuelmain_sha: 4b6bcf60d712b74a3a53252bfbb5ca34aa7497c4
nailgun_sha: 1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666
ostf_sha: f9c37d0876141e1550eb4e703a8e500cd463282f
production: docker
python-fuelclient_sha: 61431ed16fc00039a269424bdbaa410277eff609
release: '6.1'
Baremetal, Ubuntu, HA, Neutron-gre,Ceph-all, Debug, 6.1_120
Controllers:3 Computes:46
Deployment was successful, but during full rally test we got the following trace for two from 46 iterations :
14 GetResourceFailure Failed to get the resource <Stack {u'disable_rollback': True, u'description': u'No description', u'parent': None, u'stack_status_reason': u'Resource DELETE failed: InternalServerError: Request Failed: internal server error while processing your request.', u'stack_name': u'rally_stack_cqlhgfd', u'timeout_mins': None, u'creation_time': u'2015-02-17T02:59:32Z', u'links': [{u'href': u'http://172.16.45.101:8004/v1/3ef6c9df109e4061bfcf104f562d3316/stacks/rally_stack_cqlhgfd/29b0d295-dd3f-46c3-ab7b-38d996a552ca', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'parameters': {u'public_net': u'net04_ext', u'OS::stack_name': u'rally_stack_cqlhgfd', u'image': u'TestVM', u'OS::stack_id': u'29b0d295-dd3f-46c3-ab7b-38d996a552ca', u'cidr': u'11.11.11.0/24', u'flavor': u'm1.nano'}, u'stack_owner': None, u'stack_status': u'DELETE_FAILED', u'updated_time': None, u'id': u'29b0d295-dd3f-46c3-ab7b-38d996a552ca', u'template_description': u'No description'}>: ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1 Authentication required
Traceback (most recent call last):
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 77, in _run_scenario_once
method_name)(**kwargs) or scenario_output
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/stacks.py", line 74, in create_and_delete_stack
self._delete_stack(stack)
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 254, in func_atomic_actions
f = func(self, *args, **kwargs)
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/utils.py", line 117, in _delete_stack
check_interval=CONF.benchmark.heat_stack_delete_poll_interval)
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 130, in wait_for_delete
resource = update_resource(resource)
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 54, in _get_from_manager
raise exceptions.GetResourceFailure(resource=resource, err=e)
GetResourceFailure: Failed to get the resource <Stack {u'disable_rollback': True, u'description': u'No description', u'parent': None, u'stack_status_reason': u'Resource DELETE failed: InternalServerError: Request Failed: internal server error while processing your request.', u'stack_name': u'rally_stack_cqlhgfd', u'timeout_mins': None, u'creation_time': u'2015-02-17T02:59:32Z', u'links': [{u'href': u'http://172.16.45.101:8004/v1/3ef6c9df109e4061bfcf104f562d3316/stacks/rally_stack_cqlhgfd/29b0d295-dd3f-46c3-ab7b-38d996a552ca', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'parameters': {u'public_net': u'net04_ext', u'OS::stack_name': u'rally_stack_cqlhgfd', u'image': u'TestVM', u'OS::stack_id': u'29b0d295-dd3f-46c3-ab7b-38d996a552ca', u'cidr': u'11.11.11.0/24', u'flavor': u'm1.nano'}, u'stack_owner': None, u'stack_status': u'DELETE_FAILED', u'updated_time': None, u'id': u'29b0d295-dd3f-46c3-ab7b-38d996a552ca', u'template_description': u'No description'}>: ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1
Authentication required
during normal creating similar stack manually, we can see the following in heat-all.log:
<135>Feb 17 16:30:46 node-1 heat-api 2015-02-17 16:30:46.050 4306 DEBUG heat.common.serializers [req-6eab6b20-ae80-443c-a2ce-c249f34464ef None] JSON response : {"resources": [{"resource_name": "server_port", "links": [{"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108/resources/server_port", "rel": "self"}, {"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108", "rel": "stack"}], "logical_resource_id": "server_port", "resource_status": "CREATE_COMPLETE", "updated_time": "2015-02-17T15:55:34Z", "required_by": ["server"], "resource_status_reason": "state changed", "physical_resource_id": "21c3bb81-0b12-4fde-9711-ba99ef0ada80", "resource_type": "OS::Neutron::Port"}, {"resource_name": "router_interface", "links": [{"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108/resources/router_interface", "rel": "self"}, {"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108", "rel": "stack"}], "logical_resource_id": "router_interface", "resource_status": "CREATE_COMPLETE", "updated_time": "2015-02-17T15:55:40Z", "required_by": [], "resource_status_reason": "state changed", "physical_resource_id": "fdfc8411-2595-4829-84a3-a7565fd77910:subnet_id=5a0efe83-1a2c-44b0-8418-4045a65c7c99", "resource_type": "OS::Neutron::RouterInterface"}, {"resource_name": "private_subnet", "links": [{"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108/resources/private_subnet", "rel": "self"}, {"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108", "rel": "stack"}], "logical_resource_id": "private_subnet", "resource_status": "CREATE_COMPLETE", "updated_time": "2015-02-17T15:55:33Z", "required_by": ["router_interface", "server_port"], "resource_status_reason": "state changed", "physical_resource_id": "5a0efe83-1a2c-44b0-8418-4045a65c7c99", "resource_type": "OS::Neutron::Subnet"}, {"resource_name": "private_net", "links": [{"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108/resources/private_net", "rel": "self"}, {"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108", "rel": "stack"}], "logical_resource_id": "private_net", "resource_status": "CREATE_COMPLETE", "updated_time": "2015-02-17T15:55:31Z", "required_by": ["server_port", "private_subnet"], "resource_status_reason": "state changed", "physical_resource_id": "76e5162b-c1ee-4ea0-9193-a9726c1d2e90", "resource_type": "OS::Neutron::Net"}, {"resource_name": "router", "links": [{"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108/resources/router", "rel": "self"}, {"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108", "rel": "stack"}], "logical_resource_id": "router", "resource_status": "CREATE_COMPLETE", "updated_time": "2015-02-17T15:55:31Z", "required_by": ["router_interface"], "resource_status_reason": "state changed", "physical_resource_id": "fdfc8411-2595-4829-84a3-a7565fd77910", "resource_type": "OS::Neutron::Router"}, {"resource_name": "server", "links": [{"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108/resources/server", "rel": "self"}, {"href": "http://172.16.45.101:8004/v1/07eb9af0ac2e4b77b3a203087f821e43/stacks/listomin-test/cadd8c4f-004e-4018-8e69-80a3e895b108", "rel": "stack"}], "logical_resource_id": "server", "resource_status": "CREATE_COMPLETE", "updated_time": "2015-02-17T15:55:46Z", "required_by": [], "resource_status_reason": "state changed", "physical_resource_id": "e1c8f6d5-07eb-4ecc-b819-57c1d69003ae", "resource_type": "OS::Nova::Server"}]} to_json /usr/lib/python2.7/dist-packages/heat/common/serializers.py:43
We can see IDs of resources which stack created.
But if I try to find instance id for stack with name "rally_stack_cqlhgfd", I can't.
I have found the following error in nova-all.log. The error occurred at the similat time, when iteration of test was failed.
<183>Feb 17 15:55:48 node-1 nova-api RESP:401 CaseInsensitiveDict({'content-length': '23', 'www-authenticate': "Keystone uri='http://192.168.0.2:35357/v2.0'", 'connection': 'close', 'date': 'Tue, 17 Feb 2015 15:55:48 GMT', 'content-type': 'text/plain', 'x-openstack-request-id': 'req-b6499e41-afa5-4b16-9872-48ba8972c20f'}) Authentication required
I think it's nova issue.
diagnostic snapshot is here: https://drive.google.com/a/mirantis.com/file/d/0Bx4ptZV1Jt7hX0lKVHhqTW5kS0k/view?usp=sharing
Looks like the diagnostic snapshot is no longer available :( Let's try to reproduce it again.