Mirantis OpenStack

Murano show environment failure

Bug #1491558 reported by Viktoria Efimova
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Won't Fix
Low
MOS Murano
Mirantis OpenStack 8.0

Bug Description

During deletion of deployed environment rally sends GET requests on its status periodically. On one of such requests it recieves "Internal Server Error" response.

from Murano engine log we can say that stack from env was successfuly deleted:
2015-09-02 01:45:19.405 14062 DEBUG heatclient.common.http [-] curl -g -i -X GET -H 'X-Auth-Token: {SHA1}f4f49051718b8801813a194c06bcc4d2dc112060' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' http://172.16.45.102:8004/v1/323b2612cc5447a8b789b84a79f71072/stacks/murano-shyydie234ywc1d log_curl_request /usr/lib/python2.7/dist-packages/heatclient/common/http.py:128
2015-09-02 01:45:19.467 14062 DEBUG heatclient.common.http [-]
HTTP/1.1 404 Not Found
date: Wed, 02 Sep 2015 01:45:19 GMT
connection: close
content-type: application/json; charset=UTF-8
content-length: 611
x-openstack-request-id: req-fd57c11e-2c89-4f79-bd3f-e03cb00177aa

{"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Stack (murano-shyydie234ywc1d) could not be found.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/dist-packages/heat/common/context.py\", line 300, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/dist-packages/heat/engine/service.py\", line 422, in identify_stack\n raise exception.StackNotFound(stack_name=stack_name)\n\nStackNotFound: The Stack (murano-shyydie234ywc1d) could not be found.\n", "type": "StackNotFound"}, "title": "Not Found"}
 log_http_response /usr/lib/python2.7/dist-packages/heatclient/common/http.py:141
2015-09-02 01:45:19.469 14062 DEBUG murano.dsl.executor [-] 187da65abed44496b0e8997b0847a6f1: End execution: io.murano.system.HeatStack.delete (-5678443148194278433) _invoke_method_implementation /usr/lib/python2.7/dist-packages/murano/dsl/executor.py:161
2015-09-02 01:45:19.470 14062 DEBUG murano.dsl.executor [-] 187da65abed44496b0e8997b0847a6f1: End execution: io.murano.Environment.destroy (-6227509898938960988) _invoke_method_implementation /usr/lib/python2.7/dist-packages/murano/dsl/executor.py:161
2015-09-02 01:45:19.471 14062 DEBUG murano.dsl.executor [-] 187da65abed44496b0e8997b0847a6f1: Begin execution: io.murano.Object.destroy (-8418427923682810588) _invoke_method_implementation /usr/lib/python2.7/dist-packages/murano/dsl/executor.py:146
2015-09-02 01:45:19.472 14062 DEBUG murano.dsl.executor [-] 187da65abed44496b0e8997b0847a6f1: End execution: io.murano.Object.destroy (-8418427923682810588) _invoke_method_implementation /usr/lib/python2.7/dist-packages/murano/dsl/executor.py:161
2015-09-02 01:45:19.473 14062 INFO murano.common.engine [-] Invoking post-cleanup hooks
2015-09-02 01:45:19.477 14062 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is a10683100a764f688494f97ac7e7cb8c _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:311
2015-09-02 01:45:19.478 14062 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is 52761fa746294e29891f03304b17f032. _add_unique_id /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-09-02 01:45:19.480 14062 DEBUG oslo_messaging._drivers.impl_rabbit [-] Publisher.send: sending message results to {'oslo.message': '{"_unique_id": "52761fa746294e29891f03304b17f032", "_msg_id": "a10683100a764f688494f97ac7e7cb8c", "args": {"environment_id": "fb0be39dbe6c4590ba854d56d3e29e1d", "result": {"action": {"isException": false, "result": null}, "model": {"ObjectsCopy": null, "Attributes": [], "Objects": null, "SystemData": {}}}}, "method": "process_result", "_reply_q": "reply_91676eed03c64737b48411eff37dedc0"}', 'oslo.version': '2.0'} with routing key murano send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:407

from Murano api log:
2015-09-02 01:45:19.868 13071 DEBUG murano.api.middleware.version_negotiation [-] Determining version of request: GET /v1/environments/fb0be39dbe6c4590ba854d56d3e29e1d Accept: */* process_request /usr/lib/python2.7/dist-packages/murano/api/middleware/version_negotiation.py:47
2015-09-02 01:45:19.868 13071 DEBUG murano.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python2.7/dist-packages/murano/api/middleware/version_negotiation.py:49
2015-09-02 01:45:19.868 13071 DEBUG murano.api.middleware.version_negotiation [-] Matched version: v1 process_request /usr/lib/python2.7/dist-packages/murano/api/middleware/version_negotiation.py:61
2015-09-02 01:45:19.869 13071 DEBUG murano.api.middleware.version_negotiation [-] new path /v1/environments/fb0be39dbe6c4590ba854d56d3e29e1d process_request /usr/lib/python2.7/dist-packages/murano/api/middleware/version_negotiation.py:62
2015-09-02 01:45:19.871 13071 DEBUG murano.common.wsgi [-] Empty body provided in request deserialize_body /usr/lib/python2.7/dist-packages/murano/common/wsgi.py:677
2015-09-02 01:45:19.872 13071 DEBUG murano.api.v1.environments [-] Environments:Show <Id: fb0be39dbe6c4590ba854d56d3e29e1d> show /usr/lib/python2.7/dist-packages/murano/api/v1/environments.py:78
2015-09-02 01:45:19.875 13071 INFO murano.openstack.common.policy [-] Can not find policy directory: policy.d
2015-09-02 01:45:19.878 13071 AUDIT murano.common.policy [-] Policy check succeeded for rule 'show_environment' on target {'environment_id': u'fb0be39dbe6c4590ba854d56d3e29e1d'}
2015-09-02 01:45:19.991 13071 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is 7b705b09d2dc4994ad76475588bae696. _add_unique_id /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-09-02 01:45:19.992 13071 DEBUG oslo_messaging._drivers.impl_rabbit [-] Publisher.send: sending message reply_a29f50c556394f87ba5fe2c996445911 to {'oslo.message': '{"_unique_id": "7b705b09d2dc4994ad76475588bae696", "failure": null, "result": null, "_msg_id": "ad2ebc52f93d4c41ab0db92955632a2c"}', 'oslo.version': '2.0'} with routing key reply_a29f50c556394f87ba5fe2c996445911 send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:407
2015-09-02 01:45:20.000 13071 ERROR murano.api.v1.request_statistics [-] API Environments method Show raised an exception
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics Traceback (most recent call last):
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/api/v1/request_statistics.py", line 56, in wrap
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics result = func(*args, **kwargs)
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/api/v1/environments.py", line 104, in show
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics env['services'] = get_data(environment_id, '/services', session_id)
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/db/services/core_services.py", line 54, in get_data
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics env_description = get_description(environment_id, session_id)
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/db/services/environments.py", line 181, in get_environment_description
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics env_description = env.description
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics AttributeError: 'NoneType' object has no attribute 'description'
2015-09-02 01:45:20.000 13071 TRACE murano.api.v1.request_statistics
2015-09-02 01:45:20.002 13071 DEBUG murano.api.v1.request_statistics [-] Updating count stats for Environments, Show on object <murano.api.v1.request_statistics.RequestStatisticsCollection object at 0x7f7fdeb0ce50> update_error_count /usr/lib/python2.7/dist-packages/murano/api/v1/request_statistics.py:87

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Viktoria - please provide details about the env (MOS version, number of controllers / computes, etc) and exact steps to reproduce

Changed in mos:
status: New → Incomplete
assignee: nobody → Viktoria Efimova (vefimova)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Looks like some issue in Murano API which should be fixed, but it reproduced only during execution of Rally scenarios and doesn't affect users.

Changed in mos:
importance: Undecided → Low
milestone: none → 8.0
Revision history for this message
Viktoria Efimova (vefimova) wrote :

The failures happen during create_and_deploy_environment rally scenario.

Deployment:
Controllers:3, Computes:47, VLAN+DVR, 182 build

You can find snapshot at http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-03_11-48-28.tar.xz

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Murano team, could you please take a look at Viktoria's comments?

Changed in mos:
status: Incomplete → Confirmed
assignee: Viktoria Efimova (vefimova) → MOS Murano (mos-murano)
Revision history for this message
Ekaterina Chernova (efedorova) wrote :

Looks like problem with the test script

Changed in mos:
assignee: MOS Murano (mos-murano) → Anastasia Kuznetsova (akuznetsova)
Revision history for this message
Anastasia Kuznetsova (akuznetsova) wrote :

As far as I know such error was reproduces only once during cycle of Murano scale testing in 7.0 and it looks like it was an accident.
Mos-performance team didn't start cycle of scale testing for 8.0, so we do not have any new data for this bug. I propose to leave it as it is for now and update it later when it reproduces again.

Changed in mos:
assignee: Anastasia Kuznetsova (akuznetsova) → MOS Murano (mos-murano)
Revision history for this message
Serg Melikyan (smelikyan) wrote :

I don't see a reason leaving this bug in Open state, moved to Incomplete

Changed in mos:
status: Confirmed → Incomplete
Stan Lagun (slagun)
no longer affects: mos/8.0.x
ruhe (ruhe)
Changed in mos:
status: Incomplete → Won't Fix
Roman Podoliaka (rpodolyaka)
tags: added: area-murano
removed: murano
Serg Melikyan (smelikyan)
no longer affects: mos/9.0.x
Changed in mos:
status: Won't Fix → Invalid
status: Invalid → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.