Race condition while uploading packages with same tags in parallel

[Anastasia Kuznetsova]

During writing scenarios for testing performance of package creation / deletion / modification I periodically faced to a traceback:

2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog [req-1a4e849b-ea61-4059-8774-90016fd5d71e ecda7507f1f94ed88457052695babca3 7b146cf7a1af4ec88c42ee9e9a3d720c - - -] Package with specified full name is already registered
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog Traceback (most recent call last):
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/opt/stack/new/murano/murano/api/v1/catalog.py", line 243, in upload
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog package_meta, req.context.tenant)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/opt/stack/new/murano/murano/db/catalog/api.py", line 378, in package_upload
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog package.save(session)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog session.flush()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2004, in flush
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog self._flush(objects)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2122, in _flush
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog transaction.rollback(_capture_exception=True)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog compat.reraise(exc_type, exc_value, exc_tb)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2086, in _flush
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog flush_context.execute()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog rec.execute(self)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog uow
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog mapper, table, insert)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 781, in _emit_insert_statements
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog execute(statement, params)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog return meth(self, multiparams, params)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog return connection._execute_clauseelement(self, multiparams, params)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog compiled_sql, distilled_params
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog context)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog util.raise_from_cause(newraise, exc_info)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog reraise(type(exception), exception, tb=exc_tb)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog context)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog cursor.execute(statement, parameters)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 134, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog result = self._query(query)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 282, in _query
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog conn.query(q)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 768, in query
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 929, in _read_query_result
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog result.read()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1125, in read
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog first_packet = self.connection._read_packet()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 893, in _read_packet
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog packet.check_error()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 369, in check_error
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog err.raise_mysql_exception(self._data)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog _check_mysql_exception(errinfo)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog raise errorclass(errno, errorvalue)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'App' for key 'name'") [SQL: u'INSERT INTO tag (created, updated, id, name) VALUES (%s, %s, %s, %s)'] [parameters: (datetime.datetime(2015, 9, 21, 13, 46, 48, 640836), datetime.datetime(2015, 9, 21, 13, 46, 48, 640843), '4475292c00044342a8643f9e894c3fde', 'App')]
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog

It happens because I tried to import a few packages with the same tag in parallel

[Nikolay Starodubtsev (starodubcevna)]

will be the fix with retrying good enough for this issue? I mean something like we did here: https://review.openstack.org/#/c/221827/

[Anastasia Kuznetsova (akuznetsova)]

Maybe it can be fixed in the transaction layer ?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/257858

[Kirill Zaitsev (kzaitsev)]

I do not actually think that this one has the backport potential for kilo

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257858
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=8865876b1c56e2a33f3f3f3a19ee2ecd9261ffa9
Submitter: Jenkins
Branch: master

commit 8865876b1c56e2a33f3f3f3a19ee2ecd9261ffa9
Author: Nikolay Starodubtsev <email address hidden>
Date: Tue Dec 15 15:00:03 2015 +0300

    Update _get_tags function to prevent race condition

    Add `session.begin(nested=True` inside get tags to prevent duplicate
    tags creation if murano executes in amount of threads.

    Change-Id: I10968e2f2f758a5904943d273a9302a5cfb1a8c9
    Closes-Bug: #1498097

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/murano 2.0.0.0b3 development milestone.