For a multisite setup 2 Ceph clusters have been deployed:
Primary: https://paste.ubuntu.com/p/CkgMnvV6th/
Secondary: https://paste.ubuntu.com/p/gy4Zm2GCPh/
When a relation between Primary Cluster's "ceph-radosgw:primary" and Secondary Cluster's "ceph-radosgw:secondary" is added, it fails at secondary site with following error:
** Complete error at: https://paste.ubuntu.com/p/Dt6TPxz7JR/
=================================================================================================
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed Traceback (most recent call last):
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/secondary-relation-changed", line 1014, in <module>
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed hooks.execute(sys.argv)
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/hookenv.py", line 963, in execute
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed self._hooks[hook_name]()
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/secondary-relation-changed", line 922, in secondary_relation_changed
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed multisite.pull_realm(url=master_data['url'],
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/multisite.py", line 584, in pull_realm
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed return json.loads(_check_output(cmd))
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed return f(*args, **kwargs)
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-0/charm/hooks/multisite.py", line 33, in _check_output
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed return subprocess.check_output(
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed File "/usr/lib/python3.10/subprocess.py", line 524, in run
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed raise CalledProcessError(retcode, process.args,
unit-ceph-radosgw-0: 10:44:20 WARNING unit.ceph-radosgw/0.secondary-relation-changed subprocess.CalledProcessError: Command '['radosgw-admin', '--id=rgw.juju-429d53-secondary-6', 'realm', 'pull', '--url=https://10.5.0.142:443', '--access-key=CYCWO1BL6TCN1SUUM6N7', '--secret=TwWo5TYSJ1Q2WSssP3puWhKRiWgAku1H3c09KsQO']' returned non-zero exit status 5.
Realm Pull failures are pretty common when the different RGW sites are unaware of each-other's certs. This issue can be solved by using the same keystone application to provide "certificates" relation to both primary and secondary ceph-radosgw applications.
Alternatively, The Cert files can also be manually made available at the corresponding "other" rgw host.