Installation Fails on Arm64 Architecture

[Johann David Krister Andersson]

The installation process fails when running it on arm64 architecture. My guess is that is is an issue with the image not supporting arm64, although the charm seems to let itself be installed on arm64.

To recreate bug:
1. Create a VM using arm64 architecture
2. Install microk8s and juju
3. Bootstrap juju on microk8s
4. Deploy postgresql k8s: juju deploy cs:~postgresql-charmers/postgresql-k8s postgresql

juju --version
2.9.32-ubuntu-arm64

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

uname -a
Linux bug-vm-fast 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:05:39 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

juju debug-log
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install from clientrel import ClientRelations
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install File "/var/lib/juju/agents/unit-postgresql-0/charm/src/clientrel.py", line 29, in <module>
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install import pg
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install File "/var/lib/juju/agents/unit-postgresql-0/charm/src/pg.py", line 20, in <module>
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install import psycopg2
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/psycopg2/__init__.py", line 51, in <module>
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install from psycopg2._psycopg import ( # noqa
application-postgresql: 10:31:06 WARNING unit.postgresql/0.install ModuleNotFoundError: No module named 'psycopg2._psycopg'
application-postgresql: 10:31:06 ERROR juju.worker.caasoperator.uniter.postgresql/0.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1

[Johann David Krister Andersson (jdkandersson)]

juju-crashdump doesn't report anything, I'm guessing because the problem occurred during the installation process.

juju-crashdump
2022-08-08 10:37:36,679 - juju-crashdump started.
2022-08-08 10:37:38,020 - 0 machines found. No tarballs to retrieve.
2022-08-08 10:37:38,036 - juju-crashdump finished.

[Johann David Krister Andersson (jdkandersson)]

I also tried to build the charm from source on the same arm64 machine using charmcraft, and it produces the following error (it seems that either pg_config was not installed or not added to the PATH):

2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.083 Starting charm builder
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.083 Logging execution to '/tmp/charmcraft.log'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.083 Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.083 Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.083 Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 8 '/build/' => '/build\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 9 '/revision' => '/revision\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 10 '/venv' => '/venv\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 1 '/venv\n' => '/venv\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 2 '*.py[cod]\n' => '.*/[^/]*\\.py[cod]\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 3 '*.charm\n' => '.*/[^/]*\\.charm\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Building charm in '/root/parts/charm/install'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Linking in generic paths
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.084 Ignoring directory because of rules: '.git'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.085 Ignoring file because of rules: '.jujuignore'
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.087 Creating the dispatch mechanism
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.087 Creating the 'install' hook script pointing to dispatch
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.087 Creating the 'start' hook script pointing to dispatch
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 01:01:48.087 Creating the 'upgrade-charm' hook script pointing to dispatch
2022-08-08 11:01:50.490 :: 2022-08-08 01:01:50.156 :: 2022-08-08 ...

[Johann David Krister Andersson (jdkandersson)]

I resolved the build problem by installing the required library in the lxd container:
1. lxc start --project charmcraft local:charmcraft-postgresql-k8s-789630-0-0-arm64
2. lxc --project charmcraft exec local:charmcraft-postgresql-k8s-789630-0-0-arm64 apt install libpq-dev
3. lxc stop --project charmcraft local:charmcraft-postgresql-k8s-789630-0-0-arm64

This resulted in creating the charm postgresql-k8s_ubuntu-20.04-arm64.charm

[Johann David Krister Andersson (jdkandersson)]

To deploy the application: juju deploy ./postgresql-k8s_ubuntu-20.04-arm64.charm --resource postgresql-image=dataplatformoci/postgres-patroni --constraints arch=arm64

This still didn't work as (1) there were still missing libraries and (2) after that was resolved using git+https://github.com/juju/charm-helpers.git#egg=charmhelpers, importing apt_update, apt_install from charmhelpers.fetch and running apt_update and installing libpq-dev, the next problem was an exec format error when starting postgres:

Traceback (most recent call last):
  File "./src/charm.py", line 566, in <module>
    main(PostgresqlOperatorCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 784, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 316, in _on_postgresql_pebble_ready
    container.restart(self._postgresql_service)
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/model.py", line 1327, in restart
    self._pebble.restart_services(service_names) # type: ignore
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/pebble.py", line 1478, in restart_services
    return self._services_action('restart', services, timeout, delay)
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/pebble.py", line 1496, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "postgresql" (cannot start service: fork/exec /usr/bin/python3: exec format error)
----- Logs from task 0 -----
2022-08-08T04:16:41Z INFO Service "postgresql" has never been started.
----- Logs from task 1 -----
2022-08-08T04:16:41Z ERROR cannot start service: fork/exec /usr/bin/python3: exec format error

It looks like the image only supports amd64 (dataplatformoci/postgres-patroni).

[Tom Haddon (mthaddon)]

The charm location is actually just "postgresql-k8s" (see https://charmhub.io/postgresql-k8s), so your deploy command should just be `juju deploy postgresql-k8s` (typically resources are associated with the charms at release time so you shouldn't need to specify the location of the image).

From checking `juju info` it seems the postgresql-k8s charm doesn't support arm64 but juju is letting you deploy to that architecture, which seems odd: https://pastebin.ubuntu.com/p/ChzK5SqgDr/

Do you get an error from that deploy command? Here's what I get (on amd64):

mthaddon@tenaya:~$ juju deploy postgresql-k8s
Located charm "postgresql-k8s" in charm-hub, revision 4
Deploying "postgresql-k8s" from charm-hub charm "postgresql-k8s", revision 4 in channel stable on focal

And the image I'm seeing with that (from `microk8s kubectl describe pod -n pg-test postgresql-k8s-0` is `registry.hub.docker.com/postgresqlcharmers/postgresql@sha256:ed0e37fd953ebee11be270e19d59a732b4ae18aa87a9a6c02f59100a30d5ba01`.

[Johann David Krister Andersson (jdkandersson)]

I get the same messages from juju as you did on amd64:

juju deploy postgresql-k8s
Located charm "postgresql-k8s" in charm-hub, revision 4
Deploying "postgresql-k8s" from charm-hub charm "postgresql-k8s", revision 4 in channel stable on focal

The difference to, for example, mariadb is that kubernetes refuses to schedule the workload due to the architecture mismatch, although juju does allow the deployment to take place:

juju deploy charmed-osm-mariadb-k8s
Located charm "charmed-osm-mariadb-k8s" in charm-hub, revision 35
Deploying "charmed-osm-mariadb-k8s" from charm-hub charm "charmed-osm-mariadb-k8s", revision 35 in channel stable on focal

microk8s.kubectl get all -n postgres-test
NAME READY STATUS RESTARTS AGE
pod/modeloperator-7bb7bc9558-fg8p7 1/1 Running 0 2m22s
pod/postgresql-k8s-operator-0 1/1 Running 0 2m1s
pod/mariadb-k8s-operator-0 1/1 Running 0 31s
pod/mariadb-k8s-0 0/1 Pending 0 21s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/modeloperator ClusterIP 10.152.183.120 <none> 17071/TCP 2m22s
service/postgresql-k8s-operator ClusterIP 10.152.183.6 <none> 30666/TCP 2m1s
service/mariadb-k8s-operator ClusterIP 10.152.183.86 <none> 30666/TCP 31s
service/mariadb-k8s ClusterIP 10.152.183.16 <none> 3306/TCP 21s
service/mariadb-k8s-endpoints ClusterIP None <none> <none> 21s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/modeloperator 1/1 1 1 2m22s

NAME DESIRED CURRENT READY AGE
replicaset.apps/modeloperator-7bb7bc9558 1 1 1 2m22s

NAME READY AGE
statefulset.apps/postgresql-k8s-operator 1/1 2m1s
statefulset.apps/mariadb-k8s-operator 1/1 31s
statefulset.apps/mariadb-k8s 0/1 21s

microk8s.kubectl describe pod mariadb-k8s-0 -n postgres-test
Name: mariadb-k8s-0
Namespace: postgres-test
Priority: 0
Node: <none>
Labels: app.kubernetes.io/name=mariadb-k8s
                controller-revision-hash=mariadb-k8s-567cb7f56
                statefulset.kubernetes.io/pod-name=mariadb-k8s-0
Annotations: apparmor.security.beta.kubernetes.io/pod: runtime/default
                charm.juju.is/modified-version: 0
                controller.juju.is/id: dab32f23-8365-4f98-8e82-bbe21e86baec
                model.juju.is/id: 3c4626e0-2f24-481b-8147-7ad9f324ac64
                seccomp.security.beta.kubernetes.io/pod: docker/default
                unit.juju.is/id: mariadb-k8s/0
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/mariadb-k8s
Init Containers:
  juju-pod-init:
    Image: jujusolutions/jujud-operator:2.9.32
    Port: <none>
    Host Port: <none>
    Command:
      /bin/sh
    Args:
      -c
      export JUJU_DATA_DIR=/var/lib/juju
      export JUJU_TO...

[Tom Haddon (mthaddon)]

Marking as invalid in favour of https://bugs.launchpad.net/snapstore-server/+bug/1984108.