in 3.5 MAAS fails to restart due to Database error during start-up

[Jacopo Rota]

I installed MAAS 3.4. After some time I upgraded to MAAS master ec89598dea250a253526d48fa717e0cf9896298e and the upgrade went smooth.

After some time, I restarted maas and it is in a crashloop

Apr 26 12:46:15 first-eel maas-regiond[98810]: maasserver.eventloop_98810.master: [info] Calling start_up to start region process
Apr 26 12:46:16 first-eel maas-regiond[98810]: maasserver.start_up: [error] Database error during start-up
Apr 26 12:46:16 first-eel maas-regiond[98810]: Traceback (most recent call last):
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
Apr 26 12:46:16 first-eel maas-regiond[98810]: return self.cursor.execute(sql, params)
Apr 26 12:46:16 first-eel maas-regiond[98810]: psycopg2.errors.UndefinedObject: large object 30599 does not exist
Apr 26 12:46:16 first-eel maas-regiond[98810]: The above exception was the direct cause of the following exception:
Apr 26 12:46:16 first-eel maas-regiond[98810]: Traceback (most recent call last):
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/start_up.py", line 217, in start_up
Apr 26 12:46:16 first-eel maas-regiond[98810]: yield deferToDatabase(inner_start_up, master=master)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
Apr 26 12:46:16 first-eel maas-regiond[98810]: result = inContext.theWork() # type: ignore[attr-defined]
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
Apr 26 12:46:16 first-eel maas-regiond[98810]: inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
Apr 26 12:46:16 first-eel maas-regiond[98810]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
Apr 26 12:46:16 first-eel maas-regiond[98810]: return func(*args, **kw)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
Apr 26 12:46:16 first-eel maas-regiond[98810]: return func(*args, **kwargs)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
Apr 26 12:46:16 first-eel maas-regiond[98810]: result = func(*args, **kwargs)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/utils/orm.py", line 726, in call_with_connection
Apr 26 12:46:16 first-eel maas-regiond[98810]: return func(*args, **kwargs)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/utils/__init__.py", line 177, in call_with_lock
Apr 26 12:46:16 first-eel maas-regiond[98810]: return func(*args, **kwargs)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/utils/orm.py", line 771, in call_within_transaction
Apr 26 12:46:16 first-eel maas-regiond[98810]: return func_outside_txn(*args, **kwargs)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/utils/orm.py", line 574, in retrier
Apr 26 12:46:16 first-eel maas-regiond[98810]: return func(*args, **kwargs)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/usr/lib/python3.10/contextlib.py", line 79, in inner
Apr 26 12:46:16 first-eel maas-regiond[98810]: return func(*args, **kwds)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/start_up.py", line 328, in inner_start_up
Apr 26 12:46:16 first-eel maas-regiond[98810]: initialize_image_storage(node)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/bootresources.py", line 1324, in initialize_image_storage
Apr 26 12:46:16 first-eel maas-regiond[98810]: export_images_from_db(region, target_dir)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/lib/python3.10/site-packages/maasserver/bootresources.py", line 1304, in export_images_from_db
Apr 26 12:46:16 first-eel maas-regiond[98810]: cursor.execute("SELECT lo_unlink(%s)", [oid])
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 66, in execute
Apr 26 12:46:16 first-eel maas-regiond[98810]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
Apr 26 12:46:16 first-eel maas-regiond[98810]: return executor(sql, params, many, context)
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 79, in _execute
Apr 26 12:46:16 first-eel maas-regiond[98810]: with self.db.wrap_database_errors:
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/django/db/utils.py", line 90, in __exit__
Apr 26 12:46:16 first-eel maas-regiond[98810]: raise dj_exc_value.with_traceback(traceback) from exc_value
Apr 26 12:46:16 first-eel maas-regiond[98810]: File "/snap/maas/34871/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
Apr 26 12:46:16 first-eel maas-regiond[98810]: return self.cursor.execute(sql, params)
Apr 26 12:46:16 first-eel maas-regiond[98810]: django.db.utils.ProgrammingError: large object 30599 does not exist

[Jacopo Rota (r00ta)]

The content of the image-storage dir is

ubuntu@first-eel:/tmp/maas$ ls -ls /var/snap/maas/common/maas/image-storage/
total 2391196
 13896 -rw-r--r-- 1 root root 14228264 Apr 26 11:29 07ede8bfca7d5a15158fafa93db0238be100d8bde60c2f3726af7de702d775fa
 90108 -rw-r--r-- 1 root root 92265111 Apr 26 11:28 13db63fae0d32f00759d74143a5d7824307c41da57c22057335f565bae214b08
421888 -rw-r--r-- 1 root root 432009216 Apr 26 11:55 317345e4fd943ecf4b0cc87dbe3253a87ef48471d542e4a9825c5fed92766b9d
 13420 -rw-r--r-- 1 root root 13738760 Apr 26 11:28 37d409fd893ba546e37c9bffc06aeb933e44154068e24d65190e5a3ad43070a9
130144 -rw-r--r-- 1 root root 133262563 Apr 26 11:29 3cc0eddcd20d5d4b5204951f90b670230e10fa60205804c63c3dda3165706d1a
113016 -rw-r--r-- 1 root root 115721961 Apr 26 11:36 42df22f67637e12f18022afda2681cacf8358b0bb0d226f95ee91d50008322d5
   316 -rw-r--r-- 1 root root 321940 Apr 26 11:28 4467e0cc7947c8d5a787cd1cb846e7488bbf3427ee98fc7932b8453a02532de3
  1180 -rw-r--r-- 1 root root 1207316 Apr 26 11:28 497bc88ad4e5773c89a8008cfda98011bdf3a4bc764f66f2298c4a2e625d436f
 90120 -rw-r--r-- 1 root root 92280162 Apr 26 11:28 49f36b79d6da5d88c2e7719fb2c40b9ca5e385eb7c1a2ff7ef8ee605b601a14c
 13372 -rw-r--r-- 1 root root 13689608 Apr 26 11:28 4e698c97786e3334418adb4cc4f92cecd7f55741bf5b28a6e9842448ab579bb8
  1200 -rw-r--r-- 1 root root 1228336 Apr 26 11:28 5897d6ffc29e2ec670a8e3ceb3ddc2b888e17fc7f5e5d42429ebf986cdcc7501
130264 -rw-r--r-- 1 root root 133388499 Apr 26 11:29 68d9c896dd3cdf1f44c18dd6f879a2da7970176f73478be061a73f707f23ded5
   288 -rw-r--r-- 1 root root 293492 Apr 26 11:28 6a8b8d8b5ffb87d5ce10fdd2ea36afebd6b0fc16d89ac5c4bf938e2c521a8196
 11296 -rw-r--r-- 1 root root 11563656 Apr 26 11:29 6fc08565d515e5638bdd2da3a877e2354c3f73edc450a2e3775045274027fcba
128828 -rw-r--r-- 1 root root 131913934 Apr 26 11:29 705d1e050199c7d518531e7dc51690f9f7c4f57c132323d36ab701774733ffdb
128692 -rw-r--r-- 1 root root 131774499 Apr 26 11:29 77a3b00e2be9a6678c67b70bb465b12d1e314a9c6e8a04662851dd9029f3e3ed
 11352 -rw-r--r-- 1 root root 11623336 Apr 26 11:32 84d231b53134f64d0622f6b6f32da257b50092309eff06fdb9016908af5b3634
 13920 -rw-r--r-- 1 root root 14252392 Apr 26 11:29 87f5c6c94e4047767cf1082de826b04117b447b406493a9970e2fedb96d28165
423280 -rw-r--r-- 1 root root 433434624 Apr 26 11:28 98f78a7d4085dd339bab9bf4ae0fcc8ce7029f37acf929bce8872811dfe152f8
   504 -rw-r--r-- 1 root root 516068 Apr 26 11:28 9cd689c35a03f4186ca53d1bc08fbaedded33860c03381b5699d88b676483d93
   416 -rw-r--r-- 1 root root 424200 Apr 26 11:28 9ef4a36dddc584a726e2f83934996ca57bcb260b16f2c6d6bc621d8f500e2dae
     4 drwxr-xr-x 2 root root 4096 Apr 26 12:54 bootloaders
113116 -rw-r--r-- 1 root root 115824304 Apr 26 11:35 cb2ab16674ca670502af2e5b1eba1280155f2c54dbe3e736c4d54c1269de37ff
128828 -rw-r--r-- 1 root root 131912959 Apr 26 11:33 d09a5d24ba43493e1342d77f3fa491f15950f7004064d83c4547f3fd1023a6ef
 11384 -rw-r--r-- 1 root root 11655528 Apr 26 11:31 d6d363d3ce15f14336003c936d5f8bae71c07c51c8a76726fb3b6bc329e3b8a3
128692 -rw-r--r-- 1 root root 131774366 Apr 26 11:32 df081eb715e0c76ff46efe5663aa768fcd216a8f8c62ab00640e76f35277a90c
130264 -rw-r--r-- 1 root root 133388375 Apr 26 11:29...

[Jacopo Rota (r00ta)]

database dump in the attachment