subprocess.CalledProcessError: Command '['/usr/bin/rsync', '-r', '--delete', '--executability', 'files/ocf/maas/maas_stonith_plugin.py', '/usr/li b/stonith/plugins/external/maas']' returned non-zero exit status 3.

[Felipe Reyes]

When deploying hacluster on jammy the following error is raised when trying to install the stonith plugins:
unit-hacluster-0: 20:31:43 INFO unit.hacluster/0.juju-log /usr/bin/rsync -r --delete --executability files/ocf/ceph/rbd /usr/lib/ocf/resource.d/ceph/rbd
unit-hacluster-0: 20:31:43 INFO unit.hacluster/0.juju-log /usr/bin/rsync -r --delete --executability files/ocf/maas/dns /usr/lib/ocf/resource.d/maas/dns
unit-hacluster-0: 20:31:43 INFO unit.hacluster/0.juju-log /usr/bin/rsync -r --delete --executability files/ocf/maas/maas_dns.py /usr/lib/heartbeat/maas_dns.py
unit-hacluster-0: 20:31:43 INFO unit.hacluster/0.juju-log /usr/bin/rsync -r --delete --executability files/ocf/maas/maasclient/ /usr/lib/heartbeat/maasclient/
unit-hacluster-0: 20:31:43 INFO unit.hacluster/0.juju-log /usr/bin/rsync -r --delete --executability files/ocf/maas/maas_stonith_plugin.py /usr/lib/stonith/plugins/external/maas
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install Traceback (most recent call last):
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/install.real", line 760, in <module>
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install hooks.execute(sys.argv)
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install File "/var/lib/juju/agents/unit-hacluster-0/charm/charmhelpers/core/hookenv.py", line 962, in execute
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install self._hooks[hook_name]()
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/install.real", line 168, in install
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install setup_ocf_files()
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install File "/var/lib/juju/agents/unit-hacluster-0/charm/hooks/utils.py", line 1143, in setup_ocf_files
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install rsync(
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install File "/var/lib/juju/agents/unit-hacluster-0/charm/charmhelpers/core/host.py", line 510, in rsync
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install return subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode('UTF-8').strip()
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install File "/usr/lib/python3.10/subprocess.py", line 524, in run
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install raise CalledProcessError(retcode, process.args,
unit-hacluster-0: 20:31:43 WARNING unit.hacluster/0.install subprocess.CalledProcessError: Command '['/usr/bin/rsync', '-r', '--delete', '--executability', 'files/ocf/maas/maas_stonith_plugin.py', '/usr/li
b/stonith/plugins/external/maas']' returned non-zero exit status 3.

[Felipe Reyes (freyes)]

The directory /usr/lib/stonith/plugins/external is not present in a jammy environment, while it is in a focal one, the directory is created by the installation of the package cluster-glue which gets pulled into the system when corosync and others are installed, here it is the apt log for a focal system

Commandline: apt-get --assume-yes --option=Dpkg::Options::=--force-confold install crmsh corosync pacemaker ipmitool libmonitoring-plugin-perl python3-requests-oauthlib python3-libmaas
Install: corosync:amd64 (3.0.3-2ubuntu2.1), libnozzle1:amd64 (1.15-1ubuntu1, automatic), libnl-route-3-200:amd64 (3.4.0-1, automatic), resource-agents:amd64 (1:4.5.0-2ubuntu2.2, automatic), libcmap4:amd64 (3.0.3-2ubuntu2.1, automatic), libb-hooks-op-check-perl:amd64 (0.22-1build2, automatic), python3-argcomplete:amd64 (1.8.1-1.3ubuntu1, automatic), python3-async-timeout:amd64 (3.0.1-1, automatic), libmodule-runtime-perl:amd64 (0.016-1, automatic), libmath-calc-units-perl:amd64 (1.07-2, automatic), libparams-validate-perl:amd64 (1.29-3, automatic), python3-terminaltables:amd64 (3.1.0-3, automatic), libxml2-utils:amd64 (2.9.10+dfsg-5ubuntu0.20.04.2, automatic), openhpid:amd64 (3.8.0-2build4, automatic), pacemaker:amd64 (2.0.3-3ubuntu4.3), python3-parallax:amd64 (1.0.5-1, automatic), libqb-dev:amd64 (1.0.5-1, automatic), python3-macaroonbakery:amd64 (1.3.1-1, automatic), libfreeipmi17:amd64 (1.6.4-3ubuntu1.1, automatic), python3-pymongo-ext:amd64 (3.10.1-0ubuntu2, automatic), libtry-tiny-perl:amd64 (0.30-1, automatic), pacemaker-resource-agents:amd64 (2.0.3-3ubuntu4.3, automatic), libcib27:amd64 (2.0.3-3ubuntu4.3, automatic), libcrmcommon34:amd64 (2.0.3-3ubuntu4.3, automatic), libdevel-callchecker-perl:amd64 (0.008-1ubuntu1, automatic), python3-libmaas:amd64 (0.6.4-0ubuntu1), libquorum5:amd64 (3.0.3-2ubuntu2.1, automatic), python3-protobuf:amd64 (3.6.1.3-2ubuntu5, automatic), libsnmp-base:amd64 (5.8+dfsg-2ubuntu2.3, automatic), freeipmi-common:amd64 (1.6.4-3ubuntu1.1, automatic), libmodule-implementation-perl:amd64 (0.09-1, automatic), python3-colorclass:amd64 (2.2.0-2, automatic), liblrm2:amd64 (1.0.12-15, automatic), libcrmservice28:amd64 (2.0.3-3ubuntu4.3, automatic), libqb0:amd64 (1.0.5-1, automatic), net-tools:amd64 (1.60+git20180626.aebd88e-1ubuntu1, automatic), python3-pymongo:amd64 (3.10.1-0ubuntu2, automatic), libdynaloader-functions-perl:amd64 (0.003-1, automatic), libstonithd26:amd64 (2.0.3-3ubuntu4.3, automatic), ipmitool:amd64 (1.8.18-8), libconfig-tiny-perl:amd64 (2.24-1, automatic), libplumb2:amd64 (1.0.12-15, automatic), libnet1:amd64 (1.1.6+dfsg-3.1build1, automatic), python3-requests-oauthlib:amd64 (1.0.0-1.1build1), libparams-classify-perl:amd64 (0.015-1build2, automatic), libsnmp35:amd64 (5.8+dfsg-2ubuntu2.3, automatic), libstatgrab10:amd64 (0.92-2, automatic), libplumbgpl2:amd64 (1.0.12-15, automatic), libsub-name-perl:amd64 (0.26-1, automatic), python3-multidict:amd64 (4.7.3-1build1, automatic), libvotequorum8:amd64 (3.0.3-2ubuntu2.1, automatic), libnl-3-200:amd64 (3.4.0-1, automatic), crmsh:amd64 (4.2.0-2ubuntu1), python3-bson-ext:amd64 (3.10.1-0ubuntu2, automatic), python3-aiohttp:amd64 (3.6.2-1build1, automatic), python3-yarl:amd64 (1.4.2-2...

[Felipe Reyes (freyes)]

ok, here it is the issue. The charm installs pacemaker and this is the chain of dependencies in focal versus jammy

focal:
pacemaker -> pacemaker-resource-agents -> resource-agents -> cluster-glue

jammy
pacemaker -> pacemaker-resource-agents -> resource-agents-base

In jammy the package resource-agents was turned into a transitional package and the agents split into resource-agents-base and resource-agents-extra, the package resource-agents-extra depends on cluster-glue.

https://launchpad.net/ubuntu/+source/resource-agents

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-hacluster/+/840905

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-hacluster/+/840905
Committed: https://opendev.org/openstack/charm-hacluster/commit/32ab2d4bda69b7dff695ff0170d3c51cecac0b36
Submitter: "Zuul (22348)"
Branch: master

commit 32ab2d4bda69b7dff695ff0170d3c51cecac0b36
Author: Felipe Reyes <email address hidden>
Date: Fri May 6 09:55:15 2022 -0400

    Install resource-agents-extra on jammy.

    The install hooks rsync a set of scripts and one of the destinations
    is /usr/lib/stonith/plugins/external, this directory is created by the
    installation of the package cluster-glue which is a pulled as an
    indirect dependency of pacemaker, this changed on >=jammy where an
    intermediate package named resource-agents was split into
    resource-agents-base and resource-agents-extra wher the latter doesn't
    get installed and it's the one that depends on cluster-glue.

    The specific chain of dependencies are:

    focal:
    pacemaker -> pacemaker-resource-agents -> resource-agents -> cluster-glue

    jammy
    pacemaker -> pacemaker-resource-agents -> resource-agents-base

    Change-Id: Ia00061bff2ebe16d35d52b256c61243935edabba
    Closes-Bug: #1971841

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/jammy
Review: https://review.opendev.org/c/openstack/charm-hacluster/+/852577

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-hacluster/+/852577
Committed: https://opendev.org/openstack/charm-hacluster/commit/98af701cb204ec1306714eee5861c83c9cb2ad47
Submitter: "Zuul (22348)"
Branch: stable/jammy

commit 98af701cb204ec1306714eee5861c83c9cb2ad47
Author: Felipe Reyes <email address hidden>
Date: Fri May 6 09:55:15 2022 -0400

    Install resource-agents-extra on jammy.

    The install hooks rsync a set of scripts and one of the destinations
    is /usr/lib/stonith/plugins/external, this directory is created by the
    installation of the package cluster-glue which is a pulled as an
    indirect dependency of pacemaker, this changed on >=jammy where an
    intermediate package named resource-agents was split into
    resource-agents-base and resource-agents-extra wher the latter doesn't
    get installed and it's the one that depends on cluster-glue.

    The specific chain of dependencies are:

    focal:
    pacemaker -> pacemaker-resource-agents -> resource-agents -> cluster-glue

    jammy
    pacemaker -> pacemaker-resource-agents -> resource-agents-base

    Change-Id: Ia00061bff2ebe16d35d52b256c61243935edabba
    Closes-Bug: #1971841
    (cherry picked from commit 32ab2d4bda69b7dff695ff0170d3c51cecac0b36)

[Alan Baghumian (alanbach)]

Was a charm including this fix ever released? I just encountered this exact same issue on Jammy:

2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 Traceback (most recent call last):
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 File "/var/lib/juju/agents/unit-ceph-dashboard-lb-hacluster-2/charm/hooks/install.real", line 763, in <module>
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 hooks.execute(sys.argv)
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 File "/var/lib/juju/agents/unit-ceph-dashboard-lb-hacluster-2/charm/charmhelpers/core/hookenv.py", line 962, in execute
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 self._hooks[hook_name]()
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 File "/var/lib/juju/agents/unit-ceph-dashboard-lb-hacluster-2/charm/hooks/install.real", line 171, in install
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 setup_ocf_files()
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 File "/var/lib/juju/agents/unit-ceph-dashboard-lb-hacluster-2/charm/hooks/utils.py", line 1143, in setup_ocf_files
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 rsync(
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 File "/var/lib/juju/agents/unit-ceph-dashboard-lb-hacluster-2/charm/charmhelpers/core/host.py", line 510, in rsync
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 return subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode('UTF-8').strip()
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 524, in run
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 raise CalledProcessError(retcode, process.args,
2023-07-25 00:40:29 WARNING unit.ceph-dashboard-lb-hacluster/2.install logger.go:60 subprocess.CalledProcessError: Command '['/usr/bin/rsync', '-r', '--delete', '--executability', 'files/ocf/maas/maas_stonith_plugin.py', '/usr/lib/stonith/plugins/external/maas']' returned non-zero exit status 3.
2023-07-25 00:40:29 ERROR juju.worker.uniter.operation runhook.go:140 hook "install" (via explicit, bespoke hook script) failed: exit status 1
2023-07-25 00:40:29 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook

ubuntu@juju-b096f0-88-lxd-1:~$ sudo dpkg -l | grep cluster-glue

ubuntu@juju-b096f0-88-lxd-1:~$ sudo dpkg -l | grep resource-agents
ii pacemaker-resource-agents 2.1.2-1ubuntu3.1 all cluster resource manager ...

[Alan Baghumian (alanbach)]

Manually installing resource-agents-extra package let's the installation to go through.

[Alex Kavanagh (ajkavanagh)]

> Charm was deployed using:

> $ juju deploy --series focal --channel 2.4/stable hacluster ceph-dashboard-lb-hacluster

This bit confuses me a little; why are you specifying 'focal' if you want to install on jammy?

Please could you indicate what revision number of the charm got installed? A "juju status hacluster" will provide all the relevant information.

Thanks.

[Alan Baghumian (alanbach)]

@Alex

You do have a valid point! Going back to the command history, I had used channel 2.0.3 and not 2.4 (later refreshed to 2.4) and jammy is not supported on that channel:

$ juju deploy --series jammy --channel 2.0.3/stable hacluster test-hacluster
ERROR series "jammy" not supported by charm, supported series are: focal, bionic. Use --force to deploy the charm anyway.

It does not help that juju info also does not show jammy:

$ juju info hacluster
name: hacluster
publisher: OpenStack Charmers
summary: Corosync Cluster Engine - membership, messaging and quorum
description: |
  Corosync/Pacemaker
store-url: https://charmhub.io/hacluster
charm-id: BVNheXef1exOfCMnIeBV7KP901K2sXKM
supports: xenial, bionic, focal, groovy, hirsute, impish
subordinate: true
relations:
  provides:
    ha: hacluster
    nrpe-external-master: nrpe-external-master
  requires:
    juju-info: juju-info
    pacemaker-remote: pacemaker-remote
    peer-availability: juju-info
channels: |
  latest/stable: 83 2022-02-09 (83) 320kB
  latest/candidate: ↑
  latest/beta: ↑
  latest/edge: 118 2023-03-01 (118) 437kB
  2.4/stable: 120 2023-05-30 (120) 433kB
  2.4/candidate: ↑
  2.4/beta: ↑
  2.4/edge: ↑
  2.0.3/stable: 113 2022-10-04 (113) 430kB
  2.0.3/candidate: ↑
  2.0.3/beta: ↑
  2.0.3/edge: ↑
  1.1.18/stable: –
  1.1.18/candidate: –
  1.1.18/beta: –
  1.1.18/edge: 83 2022-03-04 (83) 320kB

Perhaps, it would be helpful to add a small note in the charm summary or description outlining this.

[Alex Kavanagh (ajkavanagh)]

@Alan, yes, it's a little awkward that the the charmhub/juju display is of the charm in latest/stable rather than from all possible charms. We're going top be changing that shortly (for the OpenStack charms) so that the default is more up to date and reflects jammy rather than focal.