MAAS

Importing boot resources failed

Bug #2056223 reported by Anton Troyanov on 2024-03-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	MAAS	Fix Committed	High	Alexsander de Souza	MAAS 3.6.0
	3.5	Fix Committed	High	Alexsander de Souza	MAAS 3.5.0

Bug Description

maas 1:3.5.0~beta1-16214-g.1eeea8524-0ubuntu1~22.04.1

Multiple errors reported during boot resources download. Eventually all images are synced, but this Tracebacks looks scary.

Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]: Mar 05 16:54:54 maas-deb regiond[10395]:
bootresources: [critical] Importing boot resources failed.
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 821, in worker
return target()
File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 47, in work
task()
File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 182, in doWork
task()
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 137, in wrapper
return func_in_reactor(*args, **kwargs).wait(LONGTIME)
File "/usr/lib/python3/dist-packages/crochet/_eventloop.py", line 198, in wait
result.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/usr/lib/python3/dist-packages/maasserver/bootresources.py", line 905, in eb_import
failure.trap(DatabaseLockNotHeld)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
self.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 726, in call_with_connection
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/__init__.py", line 177, in call_with_lock
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/bootresources.py", line 1028, in _import_resources_with_lock
return _import_resources_internal(notify)
File "/usr/lib/python3/dist-packages/maasserver/bootresources.py", line 990, in _import_resources_internal
successful = download_all_boot_resources(
File "/usr/lib/python3/dist-packages/maasserver/bootresources.py", line 852, in download_all_boot_resources
store.finalize(notify=notify)
File "/usr/lib/python3/dist-packages/maasserver/bootresources.py", line 647, in finalize
execute_workflow(
File "/usr/lib/python3/dist-packages/maasserver/workflow/__init__.py", line 35, in wrapper
ret = asyncio.run(func(*args, **kwargs))
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
/> File "/usr/lib/python3/dist-packages/maasserver/workflow/__init__.py", line 72, in execute_workflow
result = await _call_workflow(
File "/usr/lib/python3/dist-packages/maasserver/workflow/__init__.py", line 53, in _call_workflow
result = await workflow_fn(
File "/usr/lib/python3/dist-packages/temporalio/client.py", line 568, in execute_workflow
return await (
File "/usr/lib/python3/dist-packages/temporalio/client.py", line 1107, in result
raise WorkflowFailureError(
temporalio.client.WorkflowFailureError: Workflow execution failed
/> regiond: [info] 127.0.0.1 POST /MAAS/api/2.0/images-sync-progress/ HTTP/1.1 --> 200 OK (referrer: -; agent: maas/3.5.0~beta1/16214-g.1eeea8524/7f430d9a-112e-42db-9595-34944e299f3c)
maasserver: [error] ################################ Exception: I/O operation on closed file. ################################
maasserver: [error] Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 298, in view_atomic_with_post_commit_savepoint
return view_atomic(*args, **kwargs)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 62, in __call__
response = super().__call__(request, *args, **kwargs)
File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
response = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/piston3/resource.py", line 151, in __call__
actor, anonymous = self.authenticate(request, rm)
File "/usr/lib/python3/dist-packages/piston3/resource.py", line 128, in authenticate
if not authenticator.is_authenticated(request):
File "/usr/lib/python3/dist-packages/maasserver/api/auth.py", line 72, in is_authenticated
if self.is_valid_request(request):
File "/usr/lib/python3/dist-packages/piston3/authentication.py", line 355, in is_valid_request
post_params = request.POST
File "/usr/lib/python3/dist-packages/django/core/handlers/wsgi.py", line 102, in _get_post
self._load_post_and_files()
File "/usr/lib/python3/dist-packages/django/http/request.py", line 373, in _load_post_and_files
self._post, self._files = QueryDict(self.body, encoding=self._encoding), MultiValueDict()
File "/usr/lib/python3/dist-packages/django/http/request.py", line 338, in body
self._body = self.read()
File "/usr/lib/python3/dist-packages/django/http/request.py", line 393, in read
return self._stream.read(*args, **kwargs)
File "/usr/lib/python3/dist-packages/django/core/handlers/wsgi.py", line 34, in read
result = self.buffer + self._read_limited()
File "/usr/lib/python3/dist-packages/django/core/handlers/wsgi.py", line 28, in _read_limited
result = self.stream.read(size)
File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 193, in read
return self._wrapped.read(size)
ValueError: I/O operation on closed file

Related branches

~alexsander-souza/maas:lp2056223_to_3_5

Merged into maas:3.5

MAAS Lander: Approve on 2024-03-14

Alexsander de Souza: Approve on 2024-03-14

~alexsander-souza/maas:lp2056223_prettier_logs

Merged into maas:master

Anton Troyanov: Approve on 2024-03-14

MAAS Lander: Approve on 2024-03-13

Revision history for this message

Jerzy Husakowski (jhusakowski) wrote on 2024-03-07:

This is due to a known bug in aiohttp package (https://github.com/aio-libs/aiohttp/issues/4581). Instead of producing a traceback, MAAS should emit a friendlier message that the download will be retried, as Temporal takes care of ensuring a successful download.