"execution-list" or any other "list" command can fail in case of a parallel delete being run

[Renat Akhmerov]

For example, if we ran "execution-delete" and in parallel we're trying to run "execution-list" then it may fail with the following traceback:

2020-06-25 03:10:13.354 13016 DEBUG oslo_policy.policy [req-15bc9831-f754-4404-ad76-0f0b64b911b4 - CloudBandNetworkDirector - - -] enforce: rule="workflows:delete" creds={"domain_id": null, "is_admin": false, "is_admin_project": true, "project_domain_id": null, "project_id": "CloudBandNetworkDirector", "roles": ["clear_alarm_role", "CUD_Dashboard", "CUD_Subscriptions", "CUD_AlertRule", "CUD_NS", "CUD_Alarms", "ack_alarm_role", "read_alarm_role", "offline_access", "CUD_System_Settings", "uma_authorization", "CUD_Policy", "CUD_Executions", "CUD_Resources", "CUD_Catalog"], "service_project_domain_id": null, "service_project_id": null, "service_roles": [], "service_user_domain_id": null, "service_user_id": null, "system_scope": null, "user_domain_id": null, "user_id": null} target={"project_id": "CloudBandNetworkDirector", "user_id": null} enforce /usr/local/lib/python3.6/site-packages/oslo_policy/policy.py:913
2020-06-25 03:10:13.355 13016 DEBUG mistral.api.controllers.v2.workflow [req-15bc9831-f754-4404-ad76-0f0b64b911b4 - CloudBandNetworkDirector - - -] Delete workflow [identifier=bdddiscovervnf_73c6c963-9b89-4757-a30e-19c766dd0ce0_terminate_q9p7s, namespace=90a85cbe-f7e6-47d4-b32c-8256fb1fa76a] delete /usr/local/lib/python3.6/site-packages/mistral/api/controllers/v2/workflow.py:209
2020-06-25 03:10:13.380 13016 INFO eventlet.wsgi.server [req-15bc9831-f754-4404-ad76-0f0b64b911b4 - CloudBandNetworkDirector - - -] 192.168.0.48 "DELETE /v2/workflows/bdddiscovervnf_73c6c963-9b89-4757-a30e-19c766dd0ce0_terminate_q9p7s?namespace=90a85cbe-f7e6-47d4-b32c-8256fb1fa76a HTTP/1.1" status: 204 len: 83 time: 0.1886399
2020-06-25 03:10:13.421 13018 ERROR wsme.api [req-c5c7555d-9ef8-41dc-ae52-b9e028a4c1b7 - CloudBandNetworkDirector - - -] Server-side error: "Instance '<WorkflowExecution at 0x7ff11a748358>' has been deleted, or its row is otherwise not present.". Detail:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/mistral/api/controllers/v2/execution.py", line 446, in get_all
    **filters
  File "/usr/local/lib/python3.6/site-packages/mistral/utils/rest_utils.py", line 233, in get_all
    r.call(_get_all_function)
  File "/usr/local/lib/python3.6/site-packages/mistral/utils/rest_utils.py", line 249, in call
    return super(MistralRetrying, self).call(fn, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/tenacity/__init__.py", line 391, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.6/site-packages/tenacity/__init__.py", line 338, in iter
    return fut.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/tenacity/__init__.py", line 394, in call
    result = fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/mistral/utils/rest_utils.py", line 201, in _get_all_function
    rest_resource = resource_function(db_model)
  File "/usr/local/lib/python3.6/site-packages/mistral/api/controllers/v2/execution.py", line 64, in _get_workflow_execution_resource
    rest_utils.load_deferred_fields(wf_ex, ['params', 'input'])
  File "/usr/local/lib/python3.6/site-packages/mistral/utils/rest_utils.py", line 279, in load_deferred_fields
    hasattr(ex, f)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 285, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 721, in get
    value = self.callable_(state, passive)
  File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 422, in _load_for_state
    raise orm_exc.ObjectDeletedError(state)
sqlalchemy.orm.exc.ObjectDeletedError: Instance '<WorkflowExecution at 0x7ff11a748358>' has been deleted, or its row is otherwise not present.
: sqlalchemy.orm.exc.ObjectDeletedError: Instance '<WorkflowExecution at 0x7ff11a748358>' has been deleted, or its row is otherwise not present.
2020-06-25 03:10:13.423 13018 INFO eventlet.wsgi.server [req-c5c7555d-9ef8-41dc-ae52-b9e028a4c1b7 - CloudBandNetworkDirector - - -] 192.168.0.48 "GET /v2/executions HTTP/1.1" status: 500 len: 292 time: 0.6663501

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/740638

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740638
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=4f617856b6976244cb4bc588356a1170bcf73379
Submitter: Zuul
Branch: master

commit 4f617856b6976244cb4bc588356a1170bcf73379
Author: Renat Akhmerov <email address hidden>
Date: Mon Jul 13 15:44:36 2020 +0700

    Add a protection for list commands from parallel deletes

    * While fetching a list of object in the REST layer we now check
      if the object has already been deleted in parallel (by catching
      the corresponding exception caused by lazy loading) and don't
      include it into the result set.
    * Added the corresponding test that uses mocking and two synchronized
      threads to reproduce this corner case.
    * Minor style changes.

    Change-Id: Ia92d799a421e07542f270223c1add2aae7b32aab
    Closes-Bug: #1887357