As Jeffrey mentioned, this is apparently an incompability between MAAS 3.2 and 3.1 with PostgreSQL 14 (the supported version is PostgreSQL 12 for both, on Focal, but Jammy/pgsql 14 was used on snap-based tests). At a lower level, the issue is one of the 3 rack controllers has a commissioning script failure due to a database relation claimed not to exist, but it is present in the database. An attempt to work around it was to wait for 5 minutes after `maas init` on the 1st rack controller, so to ensure the database was completely set-up / in-place, before doing `maas init` on the 2nd/3rd rack controllers. That didn't help (the exact same issue happened with the 5-minute delay), so it didn't seem like a race condition/timing issue. I noticed the PostgreSQL version (and Ubuntu series) not being the supported ones for MAAS 3.2/3.1 (focal/pgsql12), and confirmed that the issue didn't happen on the tests with debs (on focal/pgsql12), but only in snaps (in jammy/pgsql14). Even though the distro series might (or is supposed to) be less of an issue for snaps, the database version is external to that, so it could lead to issues. I asked Jeffrey to verify that hypothesis (and test snaps on focal/pgsql12), which turned out to help. He further confirmed that the 'issue' also affected 3.2.10, i.e., it was not new/exclusive to the 3.2.11 build being tested. Marking bug tasks to MAAS 3.2 and devel as Invalid, this was related to an unsupported postgresql version, as far as we can tell. Thanks! Some analysis snippets: $ maas root rack-controllers read | jq -r '.[] | [.hostname, ."commissioning_status_name"]' [ "noma", "Passed" ] [ "sunset", "Failed" ] [ "anahuac", "Passed" ] $ maas root events query hostname=sunset ... { ... "level": "INFO", "created": "Tue, 30 Jul. 2024 16:40:59", "type": "Failed commissioning", "description": "" }, { ... "level": "INFO", "created": "Tue, 30 Jul. 2024 16:40:44", "type": "Script", "description": "50-maas-01-commissioning failed" }, { ... "level": "ERROR", "created": "Tue, 30 Jul. 2024 16:40:44", "type": "Script result lookup or storage error", "description": "sunset.maas(np3dam): commissioning script '50-maas-01-commissioning' failed during post-processing." }, { ... "level": "ERROR", "created": "Tue, 30 Jul. 2024 16:40:43", "type": "Script result lookup or storage error", "description": "Failed processing commissioning data: relation \"maasserver_podhost\" does not exist\nLINE 2: FROM maasserver_podhost\n ^\nQUERY: SELECT pod_id\n FROM maasserver_podhost\n JOIN maasserver_nodeconfig\n ON maasserver_nodeconfig.node_id = maasserver_podhost. node_id\n WHERE maasserver_nodeconfig.id = NEW.node_config_id\nCONTEXT: PL/pgSQL function interface_pod_notify() line 7 at SQL statement\n" } High-level summary from it: Failed commissioning 50-maas-01-commissioning failed sunset.maas(np3dam): commissioning script '50-maas-01-commissioning' failed *** during post-processing. *** "Failed processing commissioning data: relation \"maasserver_podhost\" does not exist LINE 2: FROM maasserver_podhost ^ QUERY: SELECT pod_id FROM maasserver_podhost JOIN maasserver_nodeconfig ON maasserver_nodeconfig.node_id = maasserver_podhost.node_id WHERE maasserver_nodeconfig.id = NEW.node_config_id CONTEXT: PL/pgSQL function interface_pod_notify() line 7 at SQL statement That is, `relation \"maasserver_podhost\" does not exist`. @ src/maasserver/triggers/websocket.py 0 CREATE OR REPLACE FUNCTION interface_pod_notify() RETURNS TRIGGER AS $$ ... *7* FROM maasserver_podhost It isn't a table in the database, but a view: $ sudo -u postgres psql -d maasdb -c '\dt' | grep 'maasserver_pod' public | maasserver_podhints | table | maas public | maasserver_podhints_nodes | table | maas public | maasserver_podstoragepool | table | maas $ sudo -u postgres psql -d maasdb -c 'select table_name from INFORMATION_SCHEMA.views;' | grep 'maasserver_pod' maasserver_podhost @ src/maasserver/dbviews.py 170 # Relationship between nodes and pods they host. 171 maasserver_podhost = dedent( ... 197 _ALL_VIEWS = { ... 200 "maasserver_podhost": maasserver_podhost, 201 } ... 21 def register_all_views(): 22 """Register all views into the database.""" 23 for view_name, view_sql in _ALL_VIEWS.items(): 24 _register_view(view_name, view_sql) It's called from db upgrade (ie, `maas init` time if the DB is not yet populated: db_need_init()) @ src/maascli/snap.py ... 651 def _finalize_init(self, mode, options): ... 663 if mode in ("region", "region+rack") and db_need_init(): 664 # When in 'region' or 'region+rack' the migrations for the database 665 # must be at the same level as this controller. 666 perform_work( 667 "Performing database migrations", 668 migrate_db, 333 def migrate_db(capture=False): 334 """Migrate the database.""" ... 338 os.path.join(os.environ["SNAP"], "bin", "maas-region"), 339 "dbupgrade", @ src/maasserver/management/commands/dbupgrade.py 19 class Command(BaseCommand): 20 help = "Upgrades database schema for MAAS regiond." ... 101 def handle(self, *args, **options): ... 139 self._perform_view_installation(database) 92 def _perform_view_installation(cls, database): 93 """Register all PL/pgSQL views. ... 99 dbviews.register_all_views()