Landscape Server update fails with database schema error

[Jeff Jones]

Setting up landscape-server (19.10.8-0ubuntu1) ...
========================================================================
Attempting schema upgrade as requested.
WARNING: this could take several minutes or more.
========================================================================
2022-04-22 15:22:35.892Z INFO landscape-setup "Bootstrapping from service.conf file ..."
2022-04-22 15:22:35.913Z INFO landscape-setup "Skipping configuration migration ..."
2022-04-22 15:22:35.914Z INFO landscape-setup "Checking Landscape databases ..."
2022-04-22 15:22:35.930Z WARNING landscape-setup "Database u'landscape-standalone-account-1' encoding is not UTF-8: u'C'"
2022-04-22 15:22:35.930Z WARNING landscape-setup "Database u'landscape-standalone-knowledge' encoding is not UTF-8: u'C'"
2022-04-22 15:22:35.930Z WARNING landscape-setup "Database u'landscape-standalone-main' encoding is not UTF-8: u'C'"
2022-04-22 15:22:35.931Z WARNING landscape-setup "Database u'landscape-standalone-package' encoding is not UTF-8: u'C'"
2022-04-22 15:22:35.931Z WARNING landscape-setup "Database u'landscape-standalone-resource-1' encoding is not UTF-8: u'C'"
2022-04-22 15:22:35.932Z WARNING landscape-setup "Database u'landscape-standalone-session' encoding is not UTF-8: u'C'"
2022-04-22 15:22:35.934Z INFO landscape-setup "Checking database schema ..."
2022-04-22 15:22:37.473Z ERROR landscape-setup "Encountered database error"
Traceback (most recent call last):
  File "/opt/canonical/landscape/canonical/landscape/scripts/standalone/plugins/database.py", line 478, in update_schema
    schema.run(args=args)
  File "/opt/canonical/landscape/canonical/landscape/scripts/schema.py", line 326, in run
    sharding.upgrade()
  File "/usr/lib/python2.7/dist-packages/storm/schema/sharding.py", line 114, in upgrade
    schema.advance(store, version)
  File "/usr/lib/python2.7/dist-packages/storm/schema/schema.py", line 187, in advance
    patch_applier.apply(version)
  File "/usr/lib/python2.7/dist-packages/storm/schema/patch.py", line 114, in apply
    module.apply(self._store)
  File "/opt/canonical/landscape/canonical/landscape/schema/patch_478/main.py", line 14, in apply
    store.execute(statement)
  File "/usr/lib/python2.7/dist-packages/storm/store.py", line 109, in execute
    return self._connection.execute(statement, params, noresult)
  File "/usr/lib/python2.7/dist-packages/storm/databases/postgres.py", line 306, in execute
    return Connection.execute(self, statement, params, noresult)
  File "/usr/lib/python2.7/dist-packages/storm/database.py", line 241, in execute
    raw_cursor = self.raw_execute(statement, params)
  File "/usr/lib/python2.7/dist-packages/storm/databases/postgres.py", line 316, in raw_execute
    return Connection.raw_execute(self, statement, params)
  File "/usr/lib/python2.7/dist-packages/storm/database.py", line 374, in raw_execute
    self._run_execution(raw_cursor, args, params, statement)
  File "/usr/lib/python2.7/dist-packages/storm/database.py", line 388, in _run_execution
    self._check_disconnect(raw_cursor.execute, *args)
  File "/usr/lib/python2.7/dist-packages/storm/database.py", line 454, in _check_disconnect
    return function(*args, **kwargs)
OperationalError: could not access file "$libdir/plpython3": No such file or directory

2022-04-22 15:22:37.478Z INFO landscape-setup "Schema configuration output:\n\nLoading site configuration...\nWARNING: PostgreSQL has max_prepared_transactions set to 0, not using two-phase commit.\nSetting up database schemas (will timeout after 86400 seconds) ...\n\nUpdate process info to support full u32 UID/GID range.\n\nMigrate to plpython3.\n"
2022-04-22 15:22:37.478Z INFO landscape-setup "Checking package database initial data ..."
2022-04-22 15:22:37.496Z INFO landscape-setup "Package database already initialized."
2022-04-22 15:22:37.497Z INFO landscape-setup "Renaming stock hash-id stores ..."
2022-04-22 15:22:37.497Z INFO landscape-setup "Stock package database not loaded, ignoring stock hash-id stores."
Job for landscape-package-search.service failed because the control process exited with error code.
See "systemctl status landscape-package-search.service" and "journalctl -xe" for details.
invoke-rc.d: initscript landscape-package-search, action "start" failed.
* landscape-package-search.service - Landscape's Package Search daemon
   Loaded: loaded (/lib/systemd/system/landscape-package-search.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2022-04-22 11:22:43 EDT; 10ms ago
  Process: 7337 ExecStartPre=/opt/canonical/landscape/landscape-schema-check (code=exited, status=1/FAILURE)

Apr 22 11:22:43 landscape systemd[1]: landscape-package-search.service: Failed with result 'exit-code'.
Apr 22 11:22:43 landscape systemd[1]: Failed to start Landscape's Package Search daemon.
dpkg: error processing package landscape-server (--configure):
 installed landscape-server package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of landscape-server-quickstart:
 landscape-server-quickstart depends on landscape-server (>= 19.10.8-0ubuntu1); however:
  Package landscape-server is not configured yet.

dpkg: error processing package landscape-server-quickstart (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 landscape-server
 landscape-server-quickstart
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Neil Anderson (neilanderson)]

This also happens for a manually deployed on-prem server. Following the instructions here: https://docs.ubuntu.com/landscape/en/ReleaseNotes19.10 gives the following result when running setup-landscape-server:

$ setup-landscape-server
2022-04-27 15:16:19.022Z INFO landscape-setup "Bootstrapping from service.conf file ..."
2022-04-27 15:16:19.032Z INFO landscape-setup "Skipping configuration migration ..."
2022-04-27 15:16:19.033Z INFO landscape-setup "Checking Landscape databases ..."
2022-04-27 15:16:19.039Z INFO landscape-setup "Checking database schema ..."
2022-04-27 15:16:19.049Z ERROR landscape-setup "Encountered database error"
Traceback (most recent call last):
  File "/opt/canonical/landscape/canonical/landscape/scripts/standalone/plugins/database.py", line 478, in update_schema
    schema.run(args=args)
  File "/opt/canonical/landscape/canonical/landscape/scripts/schema.py", line 280, in run
    load_config("schema")
  File "/opt/canonical/landscape/canonical/landscape/setup.py", line 231, in load_config
    config = setup_config(instance_type)
  File "/opt/canonical/landscape/canonical/landscape/setup.py", line 124, in read_config
    "instance-letter": chr(64 + instance)})
ValueError: unsupported format character 't' (0x74) at index 82
2022-04-27 15:16:19.049Z INFO landscape-setup "Schema configuration output:\n\nLoading site configuration...\n1\n"
2022-04-27 15:16:19.050Z INFO landscape-setup "Checking package database initial data ..."
2022-04-27 15:16:19.058Z INFO landscape-setup "Package database already initialized."
2022-04-27 15:16:19.058Z INFO landscape-setup "Renaming stock hash-id stores ..."
2022-04-27 15:16:19.064Z INFO landscape-setup "No stock file to rename"

I have checked the database server and it has the correct version of postgresql-plpython-10 installed.

[Rigoberto Sanchez (sanch1)]

I'm having exactly the same issue and unable to restart the services

[Kacper Jedrol (kjedrol-crif)]

For me, it's the same when I am freshly installing Landscape. Is it possible that someone has an older version of landscape-server and could share it with me? Especially version 19.10.7-0ubuntu1?

[Josh (jerquiaga)]

You can install specific versions of packages through apt. Example:

sudo apt-get install <package name>=<version>

So, for landscape-server 19.10.7-0ubuntu1 should be:

sudo apt-get install landscape-server=19.10.7-0ubuntu1

[IT TLS (tls4ubone)]

We had the same Problem on Update from 19.10.7 to 19.10.8.

This has solve the Problem:
sudo apt-get install --reinstall landscape-server-quickstart

Now the Landscape Server is up and running.

[Josh (jerquiaga)]

I tried IT TLS method of reinstall, and I get the following:

E: Internal Error, No file name for landscape-server-quickstart:amd64

[Kacper Jedrol (kjedrol-crif)]

Hey Josh (jerquiaga),

Unfortunately, this particular package is not available in the older version. It was probably deleted from the repo.

[Josh (jerquiaga)]

Is the landscape code available somewhere that individual contributors can take a look and propose bug fixes to the team, or is it internal only to Ubuntu? Doesn't seem like this is anywhere on the official team's radar.

[Mitch Burton (mitchburton)]

Landscape Server 19.10.8 has moved from the obsolete postgresql-plpython-10 package to postgresql-plpython3-10. The schema upgrade will fail if you do not have the plpython3 package available. Please see the warning and directions here: https://docs.ubuntu.com/landscape/en/ReleaseNotes19.10#upgrading-a-manual-installation-deployment

[Josh (jerquiaga)]

Hi Mitch Burton (mitchburton),

I've got a Landscape server running 19.10 that used the landscaper-server-quickstart package to set it up. I had seen the page you linked (which seems to imply that if you used quickstart you just do a normal apt-get upgrade). Doing so gave me the error that the original bug poster supplied. I went ahead and rolled back (yay VM snapshots!) and tried manually installing the postgresql-plpython3-10 package first, with the same result. It doesn't seem that the Release notes and warnings are correct, hence the bug report.

[Jeff Jones (chairmanjones)]

I have quickstart installed; I tried the steps outlined in #9 and the database upgrade fails with the same error.

[IT Hurrah (ithurrah)]

Same error here.... :(

[Josh (jerquiaga)]

I finally gave up, since I only have 10 machines connected to Landscape. I rebuilt a new Landscape server and re-registered the clients. Seemed like the best way forward since the devs don't seem interested in fixing this.

[Markus Ehrlicher (maeh86)]

I had the same problem today and tested many hours (many, many snapshot restores :D). Just in case, I can help with my solution: Although, I installed postgresql-plpython3-10 before updating, I got the mentioned errors. After some research, I found out, that I currently used Postgres 9.5 and the plpython3-package is only for Postgres 10 (actually the name of the package says it too). So, after migrating my DB to postgres 10, all went fine and I could update our landscape-server :)

small special hint: before updating postgres, I also needed to install the folllowing packages, so that the migration of the DB went fine:
postgresql-plpython-10
postgresql-9.5-debversion
postgresql-10-debversion