multi-zone replication doesn't work

[Andrey Grebennikov]

I deploy the following bundle
https://pastebin.ubuntu.com/p/qgt55P2Hv5/

(tried both bionic-ussuri and focal-distro origins).
There are two radosgw instances, each is backed by its own Ceph cluster.
Each RGW pull info correctly, however the sync status on master:
root@juju-66e0ee-1-lxd-0:~# radosgw-admin sync status
          realm 2ce9452b-1ec6-4507-9068-4ba929071c54 (replicated)
      zonegroup f68b626c-24b7-4659-9309-1a80e2986568 (us)
           zone c2d6f2e0-28c7-4ba7-8414-2e2147a579ba (us-east)
  metadata sync no sync (zone is master)
2021-03-26T04:43:52.100+0000 7f084db56940 0 data sync zone:a2c19e5e ERROR: failed to fetch datalog info
      data sync source: a2c19e5e-5547-4adb-b3ad-88961858629b (us-west)
                        failed to retrieve sync info: (13) Permission denied

On slave:
root@juju-66e0ee-2-lxd-0:~# radosgw-admin sync status
          realm 2ce9452b-1ec6-4507-9068-4ba929071c54 (replicated)
      zonegroup f68b626c-24b7-4659-9309-1a80e2986568 (us)
           zone a2c19e5e-5547-4adb-b3ad-88961858629b (us-west)
  metadata sync preparing for full sync
                full sync: 64/64 shards
                full sync: 0 entries to sync
                incremental sync: 0/64 shards
                metadata is behind on 64 shards
                behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]
      data sync source: c2d6f2e0-28c7-4ba7-8414-2e2147a579ba (us-east)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

I see the system use on the master but don't see on the slave.
The zones are configured correctly on both, and the slave can pull the realm successfully.

One suspicious thing, on the slave there is a complaint in the log:
2021-03-26T04:47:16.430+0000 7f21d7fe7700 5 req 4 0s :get_data_changes_log_info error reading user info, uid=TD10QYJHLSIIB6UCBZ5Z can't authenticate

This is the access key of the system user used for replication, and it is correct (that's the one created on the master.

[James Page (james-page)]

I can confirm I see the same behaviour using a slightly adapted version of the bundle.

[James Page (james-page)]

It looks likes the permissions between the two deployments are not correct.

[James Page (james-page)]

Regression tested the charms with bionic/nautilus - which worked fine:

$ sudo radosgw-admin sync status
          realm 5794ec73-4009-4b53-b1e1-904bed07186e (replicated)
      zonegroup 09bcbb11-a2bf-4c42-8320-7b94734f755a (us)
           zone f9e75695-582f-4e81-9a1c-9460d622eca3 (us-east)
  metadata sync no sync (zone is master)
      data sync source: 32efb61a-7bde-49e9-9796-b793dc971ff9 (us-west)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

$ sudo radosgw-admin sync status
          realm 5794ec73-4009-4b53-b1e1-904bed07186e (replicated)
      zonegroup 09bcbb11-a2bf-4c42-8320-7b94734f755a (us)
           zone 32efb61a-7bde-49e9-9796-b793dc971ff9 (us-west)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: f9e75695-582f-4e81-9a1c-9460d622eca3 (us-east)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

[James Page (james-page)]

In the nautilus test deployment I have running, both sites have a multisite-sync system user configured - I'm pretty sure that with the octopus deployment that slave site did not have that which is odd because there is no Octopus specific code in the charm.

[James Page (james-page)]

Confirmed - the system user is absent from the slave deployment

[James Page (james-page)]

It looks like the metadata sync is failing for some reason, resulting in the secondary zone not have the right system user - which results in the errors on the primary zone when it attempts to sync data back

[James Page (james-page)]

I don't believe this is a charm issue - it looks like a bug in the metadata sync process in ceph itself.

[Andrey Grebennikov (agrebennikov)]

"the system user is absent from the slave deployment"
can you elaborate on this? In the procedure on docs.ceph.com there is no step to configure the system user on the secondary site - it is supposed to come through the replication once I pull the realm.
And just to add once again - if I follow the procedure manually - I'm not observing this issue of improper authentication, however the metadata isn't being synced.
I'm talking https://docs.ceph.com/en/latest/radosgw/multisite/ (which actually has a few mistakes)
and https://dokk.org/documentation/ceph/v10.2.8/radosgw/multisite/ hat was a great manual.

[James Page (james-page)]

OK figured this out - the master site has an .otp pool that is configured with size 3 (and the supplied test bundle only has single OSD hosts). Setting the size of this pool to 1 resolves the issue but this does highlight that the .otp pool is not created by the charm and gets auto-created by rgw - with incorrect configuration.

[James Page (james-page)]

The otp_pool was new in >= mimic.

The radosgw daemon will create the pool if it does not exist - in a normal deployment this is created with the default size (3) which is fine and then gets autotuned as need be.

[James Page (james-page)]

Removing field high - in anything other than a cut down test deployment this will work fine - however a charm update is appropriate to get consistent pool configuration.

[James Page (james-page)]

With proposed fix applied:

$ sudo radosgw-admin sync status
          realm 964d0450-a906-4726-9c1c-a2aa5d788684 (replicated)
      zonegroup f846f9a9-c075-4541-b449-3891b796f480 (us)
           zone 8e6e56a1-cdfc-49f7-a581-1f4a8059037e (us-west)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: 1e2de205-5862-4e20-af12-7268565ad8a0 (us-east)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/786030
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/15d7a9d82729758e50efb48106f8cd1f1284210c
Submitter: "Zuul (22348)"
Branch: master

commit 15d7a9d82729758e50efb48106f8cd1f1284210c
Author: James Page <email address hidden>
Date: Tue Apr 13 10:43:39 2021 +0100

    Add otp pool to broker request

    Ceph RADOS gateway >= Mimic has an additional metadata pool (otp).

    Add this to the broker request to ensure that its created correctly
    by the ceph-mon application rather than being auto-created by the
    radosgw application

    Change-Id: I5e9b4e449bd1bc300225d223329bb62f3a381705
    Closes-Bug: 1921453