The import of images stops if there is an exception in the workflow

[Jacopo Rota]

Using 3.5.0 in HA mode, in some cases we experience the following exception while importing the images

Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: temporalio.activity: [warn] Completing activity as failed ({'activity_id': '1', 'activity_type': 'download-bootresourcefile', 'attempt': 1, 'namespace': 'default', 'task_queue': 'region-internal', 'workflow_id': 'bootresource-download:bef20bebcb87', 'workflow_run_id': '64bb0855-b5e6-43c7-970e-ef283ac16e52', 'workflow_type': 'download-bootresource'})
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: Traceback (most recent call last):
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/worker/_activity.py", line 439, in _run_activity
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: result = await impl.execute_activity(input)
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/worker/_activity.py", line 694, in execute_activity
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: return await input.fn(*input.args)
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/workflow/bootresource.py", line 145, in download_bootresourcefile
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: async for data in response.content.iter_chunked(CHUNK_SIZE):
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/aiohttp/streams.py", line 35, in __anext__
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: rv = await self.read_func()
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/aiohttp/streams.py", line 385, in read
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: await self._wait("read")
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/aiohttp/streams.py", line 304, in _wait
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: await waiter
Mar 11 14:44:51 maas-ha-2 maas-regiond[2923]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Mar 11 14:44:52 maas-ha-2 maas-regiond[2958]: regiond: [info] 127.0.0.1 POST /MAAS/api/2.0/images-sync-progress/ HTTP/1.1 --> 200 OK (referrer: -; agent: maas/3.5.0~rc1/16219-g.00477199f/b1fc96c2-d7e3-4731-91e5-5bcb893a33da)
Mar 11 14:45:02 maas-ha-2 maas-regiond[2958]: regiond: [info] 127.0.0.1 POST /MAAS/api/2.0/images-sync-progress/ HTTP/1.1 --> 200 OK (referrer: -; agent: maas/3.5.0~rc1/16219-g.00477199f/b1fc96c2-d7e3-4731-91e5-5bcb893a33da)
Mar 11 14:45:03 maas-ha-2 maas-regiond[2958]: regiond: [info] 127.0.0.1 POST /MAAS/api/2.0/images-sync-progress/ HTTP/1.1 --> 200 OK (referrer: -; agent: maas/3.5.0~rc1/16219-g.00477199f/b1fc96c2-d7e3-4731-91e5-5bcb893a33da)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: temporalio.activity: [warn] Completing activity as failed ({'activity_id': '1', 'activity_type': 'download-bootresourcefile', 'attempt': 2, 'namespace': 'default', 'task_queue': 'region-internal', 'workflow_id': 'bootresource-download:bef20bebcb87', 'workflow_run_id': '64bb0855-b5e6-43c7-970e-ef283ac16e52', 'workflow_type': 'download-bootresource'})
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: Traceback (most recent call last):
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/worker/_activity.py", line 439, in _run_activity
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: result = await impl.execute_activity(input)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/worker/_activity.py", line 694, in execute_activity
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: return await input.fn(*input.args)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/workflow/bootresource.py", line 145, in download_bootresourcefile
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: async for data in response.content.iter_chunked(CHUNK_SIZE):
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/aiohttp/streams.py", line 35, in __anext__
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: rv = await self.read_func()
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/aiohttp/streams.py", line 385, in read
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: await self._wait("read")
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: File "/snap/maas/x1/usr/lib/python3/dist-packages/aiohttp/streams.py", line 304, in _wait
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: await waiter
Mar 11 14:45:05 maas-ha-2 maas-regiond[2923]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: maasserver.bootresources: [critical] Importing boot resources failed.
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: Traceback (most recent call last):
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/usr/lib/python3.10/threading.py", line 953, in run
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: self._target(*self._args, **self._kwargs)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 821, in worker
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return target()
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 47, in work
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: task()
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 182, in doWork
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: task()
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: --- <exception caught here> ---
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: result = inContext.theWork() # type: ignore[attr-defined]
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return func(*args, **kw)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return func(*args, **kwargs)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: result = func(*args, **kwargs)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 137, in wrapper
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return func_in_reactor(*args, **kwargs).wait(LONGTIME)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/crochet/_eventloop.py", line 198, in wait
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: result.raiseException()
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: raise self.value.with_traceback(self.tb)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: current.result = callback( # type: ignore[misc]
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/bootresources.py", line 905, in eb_import
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: failure.trap(DatabaseLockNotHeld)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: self.raiseException()
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: raise self.value.with_traceback(self.tb)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: result = inContext.theWork() # type: ignore[attr-defined]
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return func(*args, **kw)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return func(*args, **kwargs)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: result = func(*args, **kwargs)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/utils/orm.py", line 726, in call_with_connection
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return func(*args, **kwargs)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/utils/__init__.py", line 177, in call_with_lock
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return func(*args, **kwargs)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/bootresources.py", line 1028, in _import_resources_with_lock
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return _import_resources_internal(notify)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/bootresources.py", line 990, in _import_resources_internal
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: successful = download_all_boot_resources(
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/bootresources.py", line 852, in download_all_boot_resources
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: store.finalize(notify=notify)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/bootresources.py", line 647, in finalize
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: execute_workflow(
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/workflow/__init__.py", line 35, in wrapper
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: ret = asyncio.run(func(*args, **kwargs))
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return loop.run_until_complete(main)
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return future.result()
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/workflow/__init__.py", line 72, in execute_workflow
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: result = await _call_workflow(
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/lib/python3.10/site-packages/maasserver/workflow/__init__.py", line 53, in _call_workflow
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: result = await workflow_fn(
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/client.py", line 568, in execute_workflow
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: return await (
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/client.py", line 1107, in result
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: raise WorkflowFailureError(
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]: temporalio.client.WorkflowFailureError: Workflow execution failed
Mar 11 14:45:05 maas-ha-2 maas-regiond[2958]:

The cli reports
maas admin boot-resources is-importing
Success.
Machine-readable output follows:
false

It's possible to force to reproduce this by adding `raise Exception("BUM")` here https://github.com/maas/maas/blob/dbd701455fa1045d7fbab45e4fc1daa139e4c6cb/src/maasserver/workflow/bootresource.py#L145

In MAAS 3.4 and below, `maas admin boot-resources is-importing` would report `True` in any case until the images are properly imported.