Patching may report error in upgrade scenario

[Al Bailey]

Brief Description
-----------------
patch controller reporting a failure on startup

Severity
--------
Major

Steps to Reproduce
------------------
Unclear: During an upgrade, restart patch controller

Expected Behavior
------------------
No errors in patch logs

Actual Behavior
----------------
Showing an error in patch logs

Reproducibility
---------------
Seen Once

System Configuration
--------------------
STD DX

Branch/Pull Time/Commit
-----------------------
stx 8.0

Last Pass
---------
Unclear. We never do upgrade tests

Timestamp/Logs
--------------
00:42:13: sw-patch-controller-daemon[53212]: patch_controller.py(733): INFO: Skipping feed dir /var/www/pages/feed/rel-21.12/ostree_repo/
2023-04-25T00:42:13: sw-patch-controller-daemon[53212]: patch_controller.py(735): INFO: Syncing /var/www/pages/feed/rel-22.12/ostree_repo/
2023-04-25T00:42:15: sw-patch-controller-daemon[53212]: patch_controller.py(748): INFO: Synced to mate feed via ostree pull: b''
2023-04-25T00:42:15: sw-patch-controller-daemon[53212]: patch_controller.py(673): INFO: patch_op_counter is: 8
2023-04-25T00:42:15: sw-patch-controller-daemon[53212]: patch_controller.py(313): INFO: sending sync complete
2023-04-25T00:42:15: sw-patch-controller-daemon[53212]: patch_controller.py(302): INFO: Handling sync complete
2023-04-25T00:42:15: sw-patch-agent[53206]: patch_agent.py(211): INFO: Handling detailed query
2023-04-25T00:42:15: sw-patch-agent[53206]: patch_agent.py(211): INFO: Handling detailed query
2023-04-25T01:19:17: sw-patch-controller-daemon[53212]: patch_controller.py(2334): ERROR: An uncaught exception has occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.9/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
  File "/usr/lib/python3/dist-packages/cgcs_patch/authapi/app.py", line 74, in __call__
    return self.v1(environ, start_response)
  File "/usr/lib/python3/dist-packages/cgcs_patch/authapi/auth_token.py", line 44, in __call__
    return super(AuthTokenMiddleware, self).__call__(env, start_response) # pylint: disable=too-many-function-args
  File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
    response = req.get_response(self._app)
  File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
  File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python3/dist-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/usr/lib/python3/dist-packages/pecan/core.py", line 840, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/usr/lib/python3/dist-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/usr/lib/python3/dist-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cgcs_patch/api/controllers/root.py", line 112, in upload
    result = pc.patch_import_api([fn])
  File "/usr/lib/python3/dist-packages/cgcs_patch/patch_controller.py", line 929, in patch_import_api
    self.base_pkgdata.loaddirs()
  File "/usr/lib/python3/dist-packages/cgcs_patch/patch_functions.py", line 203, in loaddirs
    for sw_rel in self.pkgs:
RuntimeError: dictionary changed size during iteration

Test Activity
-------------
Testing an upgrade

Workaround
----------
Manually remove a directory

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/update/+/881741

[Al Bailey (albailey1974)]

touch junk.file

sudo mkdir /var/www/pages/feed/rel-22.06
sudo sw-patch upload junk.file

sudo rmdir /var/www/pages/feed/rel-22.06
sudo sw-patch upload junk.file

Note: after the patch-controller crashes, it comes up in a clean state so it will not repeat this failure unless something else is making those folders under "feed"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/update/+/881741
Committed: https://opendev.org/starlingx/update/commit/4624457333733ed38012af8bf4abe446bddc1f04
Submitter: "Zuul (22348)"
Branch: master

commit 4624457333733ed38012af8bf4abe446bddc1f04
Author: Al Bailey <email address hidden>
Date: Thu Apr 27 14:58:05 2023 +0000

    Fix runtime patching error during upgrade

    During an upgrade, multiple patch directories may
    be detected, and the patch controller may attempt to
    iterate over those folders to determine the packages.

    The code that iterates over those folder can raise
     RuntimeError: dictionary changed size during iteration
    if the directory does not exist, but a package does.

    The fix is to iterate over the copy of the list.

    This issue only occurs in a duplex env, when the neighbor
    sync is being done. It will be encountered when a patch
    import is invoked after the older directory is removed.

    Test Plan:
       AIO-DX create multiple folders under /var/www/pages/feed
      and trigger a patch-import

    Closes-Bug: #2017916
    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: Ic78d30019bdaa93b9493bf57c7122f6de7cc6dab