init fails for improperly configured settings.DATABASES

[Adam Vaughn]

I am attempting to initialize a region+rack controller from the cli with snap version/build 3.4.0-14320-g.e23ccfa81 against a local postgres db. I can also reproduce the behavior with snap version/build
3.4.0~rc2-14317-g.15c0dc219

Step to reproduce

$ sudo maas init region+rack --database-uri postgres://maas:*removed*@localhost/maasdb --maas-url *removed*

Result

Failed to perfom migrations:ations
Traceback (most recent call last):
  File "/snap/maas/32316/bin/maas-region", line 8, in <module>
    sys.exit(run())
  File "/snap/maas/32316/lib/python3.10/site-packages/maasserver/region_script.py", line 81, in run
    run_django(is_snap, is_devenv)
  File "/snap/maas/32316/lib/python3.10/site-packages/maasserver/region_script.py", line 70, in run_django
    management.execute_from_command_line()
  File "/snap/maas/32316/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/snap/maas/32316/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/snap/maas/32316/usr/lib/python3/dist-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/snap/maas/32316/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/snap/maas/32316/lib/python3.10/site-packages/maasserver/management/commands/dbupgrade.py", line 107, in handle
    conn.ensure_connection()
  File "/snap/maas/32316/usr/lib/python3/dist-packages/django/db/backends/dummy/base.py", line 20, in complain
    raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.

I confirmed that the db string was valid by using it with psql:

$ psql postgres://maas:*removed*@localhost/maasdb
psql (14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

maasdb=>

[Jack Lloyd-Walters (lloydwaltersj)]

I'm not seeing the same error in my own attempts on 3.4.0~rc2-14317-g.15c0dc219.

Could you share the region config you're using?
It's possible it's malformed/incorrect/incomplete and is preventing the correct database settings being loaded.

[Adam Vaughn (adamvaughn)]

I cannot find an existing region config (this is a fresh install from snap)

$ ls /snap/maas/current/etc/maas/
drivers.yaml preseeds

Perhaps I am misunderstanding. Where should I find the region config?

[Adam Vaughn (adamvaughn)]

It seems that initialization can succeed if I install the snap with '--devmode'

$ sudo snap install maas --devmode
maas (3.4/stable) 3.4.0-14321-g.1027c7664 from Canonical✓ installed
$ sudo maas init region+rack --database-uri postgres://maas:*removed*@localhost/maasdb --maas-url http://*removed*:5240/MAAS
MAAS has been set up.

[Stamatis Katsaounis (skatsaounis)]

Hi Adam,

I tried to run the same init process like you and Jack and it worked fine for me:

$ sudo snap install maas --channel 3.4/stable
maas (3.4/stable) 3.4.0-14321-g.1027c7664 from Canonical✓ installed

$ sudo maas init region+rack --database-uri postgres://myuser:mypass@mypostgresserver/mydb --maas-url http://localhost:5240/MAAS
MAAS has been set up.
...

With the above init, MAAS added the database information at the regiond.conf to /var/snap/maas/current/regiond.conf:

$ sudo cat /var/snap/maas/current/regiond.conf

database_host: mypostgresserver
database_name: mydb
database_pass: mypass
database_user: myuser
maas_url: http://localhost:5240/MAAS

By the way, in case of snap, MAAS will load maasserver.djangosettings.snap which in turn will load maasserver.djangosettings.settings which will set the default driver to postgres here:

$ cat src/maasserver/djangosettings/settings.py

# 155
DATABASES = {"default": _get_default_db_config(config)}

# 85

return {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": database_name,
        "USER": database_user,
        "PASSWORD": database_pass,
        "HOST": config.database_host,
        "PORT": str(config.database_port),
        "CONN_MAX_AGE": config.database_conn_max_age,
        "OPTIONS": {
            "keepalives": int(config.database_keepalive),
            "keepalives_idle": config.database_keepalive_idle,
            "keepalives_interval": config.database_keepalive_interval,
            "keepalives_count": config.database_keepalive_count,
            "application_name": application_name,
        },
    }

In your case, you receive an exception couple of lines below:

# 166

except Exception:
    # The regiond.conf will attempt to be loaded when the 'maas' command
    # is read by a standard user. We allow this to fail and miss configure the
    # database information. Django will still complain since no 'default'
    # connection is defined.
    DATABASES = {}

As such, your DATABASES are set to Django default, which is dummy database and you receive the error. In order to reveal the error, I would like you to try the following (customize maas snap) and paste here the exception you receive:

$ snap download maas --channel 3.4/stable
$ unsquashfs maas_32469.snap
$ vim squashfs-root/lib/python3.10/site-packages/maasserver/djangosettings/settings.py

replace the above snippet (~ #166):

-except Exception:
+except Exception as e:
+ logging.getLogger(__name__).exception("Failed to read region configuration ", exc_info=e)

$ sudo snap pack ./squashfs-root
$ sudo snap install --dangerous maas_3.4.0-14321-g.1027c7664_amd64.snap
$ snap connections maas | awk '$1 != "content" && $3 == "-" {print $2}' | xargs -r -n1 sudo snap connect
$ sudo snap restart maas
$ sudo maas init region+rack --database-uri postgres://myuser:mypass@mypostgresserver/mydb --maas-url http://localhost:5240/MAAS

[Adam Vaughn (adamvaughn)]

Below is my region config. This file must have been created when I first tried to initialize maas with vault credential storage. When that failed, I had removed the vault parameters from my init command to simplify the search for the root cause, but I now see that the credentials were already stored and not overwritten by future init attempts.

database_host: localhost
database_name: maasdb
database_pass: *removed*
database_port: 5432
database_user: maas
maas_url: http://10.0.2.3:5240/MAAS
vault_approle_id: *removed*
vault_secret_id: *removed*
vault_secrets_mount: secret
vault_secrets_path: maas
vault_url: *removed*

When I remove the vault* keys from the region config, the initialization completes successfully. Below is the information on the exception thrown when reading the region config (when the vault information is included in the init command):

Failed to read region configuration
Traceback (most recent call last):
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/djangosettings/settings.py\", line 155, in <module>
 DATABASES = {\"default\": _get_default_db_config(config)}
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/djangosettings/settings.py\", line 61, in _get_default_db_config
 creds = client.get(get_db_creds_vault_path())
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/config.py\", line 25, in get_db_creds_vault_path
 assert node_id, f\"MAAS ID not set in {MAAS_ID.path}\"
AssertionError: MAAS ID not set in /var/snap/maas/common/maas/maas_id
Failed to read region configuration
Traceback (most recent call last):
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/djangosettings/settings.py\", line 155, in <module>
 DATABASES = {\"default\": _get_default_db_config(config)}
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/djangosettings/settings.py\", line 61, in _get_default_db_config
 creds = client.get(get_db_creds_vault_path())
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/config.py\", line 25, in get_db_creds_vault_path
 assert node_id, f\"MAAS ID not set in {MAAS_ID.path}\"
AssertionError: MAAS ID not set in /var/snap/maas/common/maas/maas_id
Traceback (most recent call last):
 File \"/snap/maas/x1/bin/maas-region\", line 8, in <module>
 sys.exit(run())
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/region_script.py\", line 81, in run
 run_django(is_snap, is_devenv)
 File \"/snap/maas/x1/lib/python3.10/site-packages/maasserver/region_script.py\", line 70, in run_django
 management.execute_from_command_line()
 File \"/snap/maas/x1/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 419, in execute_from_command_line
 utility.execute()
 File \"/snap/maas/x1/usr/lib/python3/dist-packages/django/core/management/__init__.py\", line 413, in execute
 self.fetch_command(subcommand).run_from_argv(self.argv)
 File \"/snap/maas/x1/usr/lib/python3/dist-packages/django/core/management/base.py\", line 354, in run_from_argv
 self.execute(*args, **cmd_options)
 File \"/snap/maas/x1/usr/lib/python3/dist-packages/django/core/management/base.py\", line 398, in execute
 output = self.handle(*args, **options)
 File \"/snap/maas/x1/l...

[Jack Lloyd-Walters (lloydwaltersj)]

Were you able to configure vault after MAAS sucessfully initialised?
the AssertionError: MAAS ID not set in /var/snap/maas/common/maas/maas_id looks like Vault tried to do something before init was completed.