Transaction error in user_creds_delete

[Thomas Herve]

Go the following error in a test build:

  File "/opt/stack/new/heat/heat/engine/stack.py", line 1528, in _delete_credentials
    self.user_creds_id)
  File "/opt/stack/new/heat/heat/objects/user_creds.py", line 67, in delete
    db_api.user_creds_delete(context, user_creds_id)
  File "/opt/stack/new/heat/heat/db/api.py", line 226, in user_creds_delete
    return IMPL.user_creds_delete(context, user_creds_id)
  File "/opt/stack/new/heat/heat/db/sqlalchemy/api.py", line 688, in user_creds_delete
    session.flush()
....
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 728, in _emit_update_statements
    (table.description, len(records), rows))
StaleDataError: UPDATE statement on table 'stack' expected to update 7 row(s); 1 were matched

In parallel the purge test was running. What's happening is that sqlalchemy resolves the foreign key and set the user_creds_id to NULL on all stacks referencing the user_creds row being deleted. But if we purge the stacks during the transaction, it fails.

It's probably worth adding a retry around that user_creds_delete call.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/285425

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/285425
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3f23ffb59224a3a923de052d924f680354222c00
Submitter: Jenkins
Branch: master

commit 3f23ffb59224a3a923de052d924f680354222c00
Author: Rabi Mishra <email address hidden>
Date: Fri Feb 26 20:12:15 2016 +0530

    Use transactions instead of session.flush

    Currently we do session.flush() after deleting objects. It
    would be better to use explicit transactions.

    This also changes update_and_save and delete to use
    'subtransaction=True' as they may part of an already ongoing
    transaction(ex. stack soft_delete in stack_delete), and the
    flush operation takes place within that transaction.

    Change-Id: I8fba68eda63e0e64e8b5e55eab92cf6a8ac5a503
    Closes-Bug: #1545577

[Rabi Mishra (rabi)]

This still seems to be happening and now it's trying to do a rollback and fails with a similar error. It seems to me like something to do with backrefs and cascade-update. Some stack objects seem to be flushed from the session between get and delete.

We may have to investigate more.

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 490, in __exit__
    self.rollback()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 487, in __exit__
    self.commit()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 392, in commit
    self._prepare_impl()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019, in flush
    self._flush(objects)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2137, in _flush
    transaction.rollback(_capture_exception=True)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/291994
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=45722e6cb448eb7a469d9910e495925842b58fc1
Submitter: Jenkins
Branch: master

commit 45722e6cb448eb7a469d9910e495925842b58fc1
Author: Rabi Mishra <email address hidden>
Date: Sat Mar 12 12:17:52 2016 +0530

    Add retries to avoid dberror for user_creds_delete

    Add retries for user_creds_delete.

    Also adds cascade_backrefs=False to the relationship
    between user_creds and stack.

    Change-Id: I0e242c008d3368d30708b52f83f998b77aff5f35
    Closes-Bug: #1545577

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/296222

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/296222
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=368491cd582e6895fa9844e5e3faf770625594da
Submitter: Jenkins
Branch: stable/mitaka

commit 368491cd582e6895fa9844e5e3faf770625594da
Author: Rabi Mishra <email address hidden>
Date: Sat Mar 12 12:17:52 2016 +0530

    Add retries to avoid dberror for user_creds_delete

    Add retries for user_creds_delete.

    Also adds cascade_backrefs=False to the relationship
    between user_creds and stack.

    Change-Id: I0e242c008d3368d30708b52f83f998b77aff5f35
    Closes-Bug: #1545577
    (cherry picked from commit 45722e6cb448eb7a469d9910e495925842b58fc1)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/302565

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302565
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=eeed79b7a2da4a8f7fcbe0c8a7c7a069f304ae11
Submitter: Jenkins
Branch: master

commit bea576f802f993204aaaaea82812820f2f51c668
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Wed Mar 30 16:09:29 2016 +0300

    Sync integration tests requirements

    The patch that auto-generates integration tests requirements [1] missed
    Mitaka release.

    This patch manually syncs the integration tests requirements hopefully
    for the last time.

    [1] Ie79338cc10cc101fbf15b51c7923e3a7b8e4fbb4

    Change-Id: I3f001bcb7900f6d822055bf3567773b3a594227d

commit 950505d956e99bc9eed7d6fb0223800251d88b95
Author: Rabi Mishra <email address hidden>
Date: Wed Mar 30 09:34:36 2016 +0530

    Revert "Check RBAC policy for nested stacks"

    This is causing issues with tripleo failures
    with ValueError, when trying do stack-preview as
    part of policy enforcement for stack DELETE.

    This reverts commit a8accbba98b5fc15562864030f49d20570bf8412.

    Change-Id: I366316f27f24c650bd435e67dd17abd8676cedf4
    Closes-Bug: #1561172
    (cherry picked from commit 354f1bcef038340d95f7dc4bc29268c9efed642e)

commit 184b09a40672ccfe1964bbf4fc72234d4cd012b6
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Mar 30 06:08:07 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I656cedd8162b5a7d1682a33ab9b836ba25d22de9

commit 0c407b8596499466b93d5b511e19b0de130b6f03
Author: Zane Bitter <email address hidden>
Date: Mon Mar 28 17:19:47 2016 -0400

    Add translation rule to delete ssh auth key from Magnum baymodel

    Add a translation rule to ensure that users with existing templates will
    not have them break when they update due to the 'ssh_authorized_key'
    property having been removed from Magnum and hidden in Heat.

    Change-Id: Id73bace93bfc2379e80777da7dc0c0bb059d6cd3
    Related-Bug: #1560284
    (cherry picked from commit 0ab68f891348eeafa1efa219070ef39f776cb6c6)

commit db06ee4514b0cc8037f037788c34631c272cfc6b
Author: Sergey Kraynev <email address hidden>
Date: Thu Mar 24 04:30:39 2016 -0400

    Increase timeout of WaitCondition due to ConcurrentTransactions

    Currently we have retry logic with random period of delay.
    There is a chance, when we have several re-try attemts with maximum
    timeout, so it can be a root cause of floating fails, when we have not
    enough time for getting all signals.

    Also decrease number of signals from 30 to 25, due to small risk, when
    one of signals goes to ConcurrentTransaction error.

    Note, that all these changes are made in corresponding functional test
    and don't affect main code.

    Change-Id: I9ca85c4378d0f62e316f4f40ff73597ffaaabadf
    Related-Bug: #1497274
    (cherry picked from commit 8f370d6174c7574c6ee611b5089eac8a4c37dbf2)

commit 145be139cc8a34c57f84e1991031cfc2595477ba
Author: OpenStack Proposal Bot <openstack-infra...

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/heat 7.0.0.0b1 development milestone.