Murano health check fails if test record persists

Bug #1411701 reported by Charlie Ott
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Charlie Ott

Bug Description

When my murano rabbitmq service stopped working a health check was run. I believe this check created a record like so:

'INSERT INTO environment (created, updated, id, name, tenant_id, version, description, networking) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2015, 1, 16, 0, 11, 28, 324036), datetime.datetime(2015, 1, 16, 0, 11, 28, 320557), 'd2e71ba446294a4182b986ef12055488', 'ost1_test-Murano_env01', '0dc5c66d16b04d48b07c868cc195f46a', 0, '{}', '{}')

However, since the rabbitmq server was down, the record was never deleted and the health check initially failed.

Later I fixed rabbitmq service issue (rabbitmq_userid/pw). The health check was run a second time, and failed again because the record already existed with the same unique id...

2015-01-16 00:11:28 INFO
eventlet.wsgi [-] 10.10.20.6 - - [16/Jan/2015 00:11:28] "POST /v1/environments HTTP/1.1" 500 4657 0.155735
2015-01-16 00:11:28 INFO
eventlet.wsgi [-] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 389, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/muranoapi/openstack/common/wsgi.py", line 197, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/muranoapi/openstack/common/wsgi.py", line 197, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/muranoapi/openstack/common/wsgi.py", line 384, in __call__
    action_result = self.execute_action(action, request, **action_args)
  File "/usr/lib/python2.6/site-packages/muranoapi/openstack/common/wsgi.py", line 398, in execute_action
    return self.dispatch(self.controller, action, request, **action_args)
  File "/usr/lib/python2.6/site-packages/muranoapi/openstack/common/wsgi.py", line 407, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/muranoapi/api/v1/request_statistics.py", line 55, in wrap
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/muranoapi/api/v1/environments.py", line 52, in create
    request.context.tenant)
  File "/usr/lib/python2.6/site-packages/muranoapi/db/services/environments.py", line 112, in create
    unit.add(environment)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 449, in __exit__
    self.commit()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 361, in commit
    self._prepare_impl()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 340, in _prepare_impl
    self.session.flush()
  File "/usr/lib/python2.6/site-packages/muranoapi/openstack/common/db/sqlalchemy/session.py", line 458, in _wrap
    _raise_if_duplicate_entry_error(e, self.bind.dialect.name)
  File "/usr/lib/python2.6/site-packages/muranoapi/openstack/common/db/sqlalchemy/session.py", line 397, in _raise_if_duplicate_entry_error
    raise exception.DBDuplicateEntry(columns, integrity_error)
DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry '0dc5c66d16b04d48b07c868cc195f46a-ost1_test-Murano_env01' for key 'tenant_id'") 'INSERT INTO environment (created, updated, id, name, tenant_id, version, description, networking) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2015, 1, 16, 0, 11, 28, 324036), datetime.datetime(2015, 1, 16, 0, 11, 28, 320557), 'd2e71ba446294a4182b986ef12055488', 'ost1_test-Murano_env01', '0dc5c66d16b04d48b07c868cc195f46a', 0, '{}', '{}')
2015-01-16 00:11:28 INFO
muranoapi.openstack.common.db.sqlalchemy.session [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
2015-01-16 00:11:28 INFO
urllib3.connectionpool [-] Starting new HTTP connection (1): 10.10.20.2
2015-01-16 00:11:28 INFO
urllib3.connectionpool [-] Starting new HTTP connection (1): 10.10.20.2
2015-01-16 00:11:28 INFO
keystoneclient.middleware.auth_token [-] Auth Token confirmed use of v2.0 apis
2015-01-16 00:11:28 INFO
urllib3.connectionpool [-] Starting new HTTP connection (1): 10.10.20.2
2015-01-16 00:11:28 INFO
eventlet.wsgi [-] (6812) accepted ('10.10.20.6', 50217)

Consider either ensuring test record is deleted before running every test, or generate a unique id each time a test is performed. likely the same ID is used so it can be easily removed after the test is successful?

Changed in fuel:
status: New → Confirmed
assignee: nobody → Tatyanka (tatyana-leontovich)
milestone: none → 6.1
Changed in fuel:
importance: Undecided → Medium
Changed in fuel:
assignee: Tatyanka (tatyana-leontovich) → Timur Nurlygayanov (tnurlygayanov)
tags: added: murano
Changed in fuel:
status: Confirmed → Incomplete
assignee: Timur Nurlygayanov (tnurlygayanov) → Charlie Ott (charlieott)
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Could you tell, what version of MOS you use?

OSTF test should generate random name for all components of environment include environment.
We need more information about your MOS installation.

Dmitry Pyzhov (dpyzhov)
tags: added: module-ostf
removed: ostf
Revision history for this message
Charlie Ott (charlieott) wrote :

i think at the time of this post we were using version 5.1. I'm away from work right now, but will respond to this with more information when i return to work next week.

Changed in fuel:
milestone: 6.1 → 7.0
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Moved to invalid, as this issue was not updated for more than 3 weeks.

Changed in fuel:
status: Incomplete → Invalid
tags: added: area-ostf
removed: module-ostf
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.