Multisite Sync "0 ERROR: failed to fetch datalog info" when migrating from a single site to multisite

[utkarsh bhatt]

Procedure Executed:
  > Expose and consume required interfaces for multisite relation.
  > Add multisite config on Primary and Secondary sites.
  > Add Primary:Secondary relation.

Juju Model Status:
  > Primary Site: https://paste.ubuntu.com/p/zrCxFXbCxN/
  > Secondary Site: https://paste.ubuntu.com/p/XKKwQ4tjNw/

Charm Errors: None

Behavioural Exception: Sync failure (shown below)

  > Primary Sync Status:
ubuntu@juju-30a478-zaza-5c5d4eebb304-3:~$ sudo radosgw-admin --id=rgw.$hn sync status
          realm 2b3b3890-f371-4516-9e38-0bea40c82026 (demo)
      zonegroup f23da673-0898-42f8-a58f-b32575e23887 (demo_zg)
           zone 559d53cd-1340-46b3-9e38-395a9198c079 (demo_mz)
  metadata sync no sync (zone is master)
2023-01-03T15:15:03.398+0000 7f2fad964e40 0 ERROR: failed to fetch datalog info
      data sync source: 1515c2fe-2afd-4527-a034-e66b813d2101 (demo_sz)
                        failed to retrieve sync info: (13) Permission denied

  > Secondary Sync Status:
ubuntu@juju-137848-secondary-6:~$ sudo radosgw-admin --id=rgw.$hn sync status
          realm 2b3b3890-f371-4516-9e38-0bea40c82026 (demo)
      zonegroup f23da673-0898-42f8-a58f-b32575e23887 (demo_zg)
           zone 1515c2fe-2afd-4527-a034-e66b813d2101 (demo_sz)
  metadata sync failed to read sync status: (2) No such file or directory
      data sync source: 559d53cd-1340-46b3-9e38-395a9198c079 (demo_mz)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

[utkarsh bhatt (utkarshbhatthere)]

Logs From Primary (Older 19th jan):
2023-01-19T00:00:06.352+0000 7fac837f6640 0 RGW-SYNC:data:sync:shard[123]: ERROR: failed to read remote data log info: ret=-5
2023-01-19T00:00:06.528+0000 7fac837f6640 0 RGW-SYNC:data:sync:shard[92]: ERROR: failed to read remote data log info: ret=-5
2023-01-19T00:00:06.568+0000 7fac837f6640 0 RGW-SYNC:data:sync:shard[119]: ERROR: failed to read remote data log info: ret=-5
2023-01-19T00:00:06.660+0000 7fac837f6640 0 RGW-SYNC:data:sync:shard[113]: ERROR: failed to read remote data log info: ret=-5
2023-01-19T00:00:06.776+0000 7fac837f6640 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5
2023-01-19T00:00:06.880+0000 7fac837f6640 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5
2023-01-19T00:00:07.008+0000 7fac837f6640 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5
2023-01-19T00:00:07.040+0000 7fac837f6640 0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5

Logs from Secondary (Datalog still not created on 08th Feb):
ceph-client.rgw.juju-137848-secondary-6.log:98293:2023-02-08T17:19:49.489+0000 7f60f7ff7640 0 rgw rados thread: ERROR: failed to fetch datalog info
ceph-client.rgw.juju-137848-secondary-6.log:98325:2023-02-08T17:20:09.489+0000 7f60f7ff7640 0 rgw rados thread: ERROR: failed to fetch datalog info
ceph-client.rgw.juju-137848-secondary-6.log:98357:2023-02-08T17:20:29.489+0000 7f60f7ff7640 0 rgw rados thread: ERROR: failed to fetch datalog info
ceph-client.rgw.juju-137848-secondary-6.log:98388:2023-02-08T17:20:49.489+0000 7f60f7ff7640 0 rgw rados thread: ERROR: failed to fetch datalog info