Cinder

cgsnapshot_id needs to be cleared before snapshot.save

Bug #1662684 reported by Xing Yang on 2017-02-07

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	High	Xing Yang

Bug Description

When a driver raises an exception in create_cgsnapshot, create_cgsnapshot in manager.py blows up. This is because cgsnapshot_id is added to the snapshot object when passing to the driver and it is cleared if the driver returns successfully. If driver raises, however, cgsnapshot_id is still in the snapshot object and the snapshot.save in the exception block throws an exception because cgsnapshot does not really exists in the db. This is another case related to the CG migration changes.

2017-02-07 10:28:37.293 ERROR oslo_messaging.rpc.server [req-3d34e844-e8c3-4b4c-bbbf-b5d6f30b9df4 admin None] Exception during message handling
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 3840, in create_group_snapshot
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server snapshot.save()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/objects/snapshot.py", line 211, in save
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server db.snapshot_update(self._context, self.id, updates)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/api.py", line 481, in snapshot_update
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server return IMPL.snapshot_update(context, snapshot_id, values)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 251, in wrapper
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 196, in wrapper
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 3128, in snapshot_update
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server return snapshot_ref
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 553, in __exit__
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server self.rollback()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 550, in __exit__
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server self.commit()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 455, in commit
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server self._prepare_impl()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 435, in _prepare_impl
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server self.session.flush()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2080, in flush
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server self._flush(objects)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2198, in _flush
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server transaction.rollback(_capture_exception=True)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2162, in _flush
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server flush_context.execute()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server rec.execute(self)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server uow
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server mapper, table, update)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 710, in _emit_update_statements
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server execute(statement, multiparams)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server return meth(self, multiparams, params)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server return connection._execute_clauseelement(self, multiparams, params)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server compiled_sql, distilled_params
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server context)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server context)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server cursor.execute(statement, parameters)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 166, in execute
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server result = self._query(query)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 322, in _query
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server conn.query(q)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 835, in query
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1019, in _read_query_result
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server result.read()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1302, in read
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server first_packet = self.connection._read_packet()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 981, in _read_packet
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server packet.check_error()
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in check_error
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server raise errorclass(errno, errval)
2017-02-07 10:28:37.293 TRACE oslo_messaging.rpc.server DBReferenceError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`cinder`.`snapshots`, CONSTRAINT `snapshots_ibfk_1` FOREIGN KEY (`cgsnapshot_id`) REFERENCES `cgsnapshots` (`id`))') [SQL: u'UPDATE snapshots SET updated_at=%(updated_at)s, cgsnapshot_id=%(cgsnapshot_id)s, status=%(status)s WHERE snapshots.id = %(snapshots_id)s'] [parameters: {'status': u'error', 'snapshots_id': u'0904278c-7d2f-4106-9d31-d126a9454b71', 'updated_at': datetime.datetime(2017, 2, 7, 15, 28, 37, 287493), 'cgsnapshot_id': 'b2547490-d62b-48b2-9c15-633959887e95'}]

Tags:

Xing Yang (xing-yang) on 2017-02-07

Changed in cinder:
assignee:	nobody → Xing Yang (xing-yang)
status:	New → Confirmed
importance:	Undecided → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-07: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/430449

Changed in cinder:
status:	Confirmed → In Progress

Revision history for this message

Xing Yang (xing-yang) wrote on 2017-02-07:

Also "snapshot.cgsnapshot" is not accessible in create/delete cgsnapshot and "volume.consistencygroup.volumes" is accessible in create volume.

tags:

added: cg

Revision history for this message

Xing Yang (xing-yang) wrote on 2017-02-07:

Some other related issues are tracked by https://bugs.launchpad.net/cinder/+bug/1661393

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-08: Fix merged to cinder (master)

Reviewed: https://review.openstack.org/430449
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=702868fd00e6b99d7ae5bda64d3410b2e5f9ddc5
Submitter: Jenkins
Branch: master

commit 702868fd00e6b99d7ae5bda64d3410b2e5f9ddc5
Author: xing-yang <email address hidden>
Date: Sat Dec 3 14:06:44 2016 -0500

Remove cgsnapshot_id before snapshot.save

    There are a few more places where information is missing for
    consistencygroup or cgsnapshot due to the CG migration changes.
    This patch tries to fix them.

    cgsnapshot_id should be removed from the snapshot object before
    snapshot.save is called in the exception block because cgsnapshot
    does not exist in the db.

cgsnapshot object is added to the snapshot object when snapshots
are passed to the driver in create/delete cgsnapshot.

Change-Id: I253a98b8a2058a00c2c577927c0b701c2f88c81b
Closes-Bug: #1662684

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-13: Fix proposed to cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/433185

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-14: Fix merged to cinder (stable/ocata)

Reviewed: https://review.openstack.org/433185
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b069c027ab3d4e0b105231f5255fcd65b4edfd90
Submitter: Jenkins
Branch: stable/ocata

commit b069c027ab3d4e0b105231f5255fcd65b4edfd90
Author: xing-yang <email address hidden>
Date: Sat Dec 3 14:06:44 2016 -0500

Remove cgsnapshot_id before snapshot.save

    There are a few more places where information is missing for
    consistencygroup or cgsnapshot due to the CG migration changes.
    This patch tries to fix them.

    cgsnapshot_id should be removed from the snapshot object before
    snapshot.save is called in the exception block because cgsnapshot
    does not exist in the db.

cgsnapshot object is added to the snapshot object when snapshots
are passed to the driver in create/delete cgsnapshot.

    Change-Id: I253a98b8a2058a00c2c577927c0b701c2f88c81b
    Closes-Bug: #1662684
    (cherry picked from commit 702868fd00e6b99d7ae5bda64d3410b2e5f9ddc5)