module 'maas' has no attribute 'MAASHelper'

[Marian Gasparovic]

Deployed hacluster with with below definitions

  maas_url: &maas-url "http://192.168.210.4:5240/MAAS"
  maas_credentials: &maas-credentials "*censored*"
  stonith_enabled: &stonith-enabled "True"

  hacluster-masakari:
    charm: cs:hacluster
    bindings:
      pacemaker-remote: *ceph-public-space
    options:
      maas_url: *maas-url
      maas_credentials: *maas-credentials
      stonith_enabled: *stonith-enabled

It resulted in a unit in error, log reveals this

2020-05-28 11:44:18 INFO juju-log Corosync and Pacemaker are validated as up and running.
2020-05-28 11:44:29 DEBUG juju-log Pacemaker is ready
2020-05-28 11:44:29 DEBUG juju-log Applying global cluster configuration
2020-05-28 11:44:30 DEBUG juju-log Configuring no-quorum-policy to stop
2020-05-28 11:44:31 DEBUG juju-log Configuring cluster-recheck-interval to 60 seconds
2020-05-28 11:44:32 DEBUG juju-log Checking monitor host configuration
2020-05-28 11:44:33 INFO juju-log Enabling STONITH for all nodes in cluster.
2020-05-28 11:44:33 DEBUG config-changed Traceback (most recent call last):
2020-05-28 11:44:33 DEBUG config-changed File "/var/lib/juju/agents/unit-hacluster-masakari-0/charm/hooks/config-changed", line 660, in <module>
2020-05-28 11:44:33 DEBUG config-changed hooks.execute(sys.argv)
2020-05-28 11:44:33 DEBUG config-changed File "/var/lib/juju/agents/unit-hacluster-masakari-0/charm/charmhelpers/core/hookenv.py", line 943, in execute
2020-05-28 11:44:33 DEBUG config-changed self._hooks[hook_name]()
2020-05-28 11:44:33 DEBUG config-changed File "/var/lib/juju/agents/unit-hacluster-masakari-0/charm/hooks/config-changed", line 200, in config_changed
2020-05-28 11:44:33 DEBUG config-changed configure_stonith()
2020-05-28 11:44:33 DEBUG config-changed File "/var/lib/juju/agents/unit-hacluster-masakari-0/charm/hooks/utils.py", line 530, in configure_stonith
2020-05-28 11:44:33 DEBUG config-changed nodes = maas.MAASHelper(url, creds).list_nodes()
2020-05-28 11:44:33 DEBUG config-changed AttributeError: module 'maas' has no attribute 'MAASHelper'
2020-05-28 11:44:33 ERROR juju.worker.uniter.operation runhook.go:132 hook "config-changed" failed: exit status 1

[Alex Kavanagh (ajkavanagh)]

Hi, please could you include the version of the charm and the version of Ubuntu? A juju status output + logs would also be very useful for the affected unit.

Thanks.

[Marian Gasparovic (marosg)]

$ juju status|grep hacluster
hacluster-masakari error 3 hacluster jujucharms 68 ubuntu
  hacluster-masakari/1 error idle 10.0.2.165 hook failed: "config-changed"
  hacluster-masakari/0* error idle 10.0.2.139 hook failed: "config-changed"
  hacluster-masakari/2 error idle 10.0.2.180 hook failed: "config-changed"

stein + bionic deployment

I will run juju-crashdump and attach

[Marian Gasparovic (marosg)]

Adding unit log. crashdump would be too big just for this

[Alex Kavanagh (ajkavanagh)]

It looks like something is up with the path and a recent change with adding python3-libmaas which is shadowing the local 'maas.py' file, which is causing the MAASHelper class to be 'missing'.

[Alex Kavanagh (ajkavanagh)]

So the issue is a bit more complicated:

There are two client library packages / modules:

* python3-maas-client ... upto eoan
* python3-libmaas ... from bionic

The hacluster charm now installs BOTH packages. Clearly this will break on focal as the python3-maas-client doesn't exist on focal. It does exist as a snap, but hacluster doesn't support installing things from snaps, and then there's the whole "closed" network installed issue where the snap might not be available.

The python3-libmaas is a more modern client library. the hacluster charm needs to support xenial (and trusty for series upgrade to xenial) and so the python3-maas-client can't be eliminated.

So the strategy going forwards can be:

1. From focal, DON'T install the python3-maas-client as it will break things.
2. Ensure that any usage of it in focal, uses the python3-libmaas library instead.

Then to solve the actual bug:

Rename the local maas library to be in a local 'lib' directory or similar. This way it can be reached at "lib.maas" which won't conflict with the system installed version of 'maas'. This is the best solution until we can package classic charms in a way such that they don't need the system packages at all.

[Alex Kavanagh (ajkavanagh)]

Marian, please could you include the bundle that was used to deploy (or the exact deploy command that was used for the charm) and the logs when reporting errors like this.

It really does make it much easier to work out what's going on in a complex system. e.g. here, it would be useful to know which version of masakari, which version of packages is being installed, etc, and this all comes from the bundle (and thus config options). The logs would also indicate what is going on to the box, and any errors that might be happening in the payload software.

As hacluster is a subordinate, knowing what else is on the same unit lends light to understanding the nature of the bug. Thank you.

[Marian Gasparovic (marosg)]

Here is the bundle. I will send crashdump when it is ready

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/733128

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/733128
Committed: https://git.openstack.org/cgit/openstack/charm-hacluster/commit/?id=eaaa5e5bd8d154016eb785c6655962633217213b
Submitter: Zuul
Branch: master

commit eaaa5e5bd8d154016eb785c6655962633217213b
Author: Liam Young <email address hidden>
Date: Wed Jun 3 12:44:39 2020 +0000

    Remove old configure_stonith

    The old configure_stonith last worked on xenial. This patch bypassies
    it given it is broken and it is interfering with the new pacemaker
    remote stonith resource setup. The code will be removed by a follow
    up patch. This patch is designed to be small for easy cherry picking.

    Change-Id: I2a704c1221bda242caaa5e87849d9984db3c6b71
    Partial-Bug: #1881114

[Aurelien Lourot (aurelien-lourot)]

This was fixed by the patch above, by turning the faulty code into dead code.

Now I have removed that dead code here: https://review.opendev.org/c/openstack/charm-hacluster/+/778626

[Aurelien Lourot (aurelien-lourot)]

So this is actually fixed since 20.08. Doing some more clean-up right now and adding to the release notes.

[Aurelien Lourot (aurelien-lourot)]

Some more dead code removal: https://review.opendev.org/c/openstack/charm-hacluster/+/778663

[Aurelien Lourot (aurelien-lourot)]

While fixing this bug, the `stonith_enabled` config option has been deprecated by blocking the unit if used. [0] A newer patch [1] now officially deprecates this config option in config.yaml. Adding it now to the release notes. [2]

[0] https://review.opendev.org/c/openstack/charm-hacluster/+/733128
[1] https://review.opendev.org/c/openstack/charm-hacluster/+/778626
[2] https://review.opendev.org/c/openstack/charm-guide/+/778667