In multisite environment, objects containing "/" in the name are not synced

[Giuseppe Petralia]

Steps to reproduce:
===================

1. Create new object on primary zone in "test-sync-bucket" callend "obj-no-slash"

2. Object is synced correctly

From secondary:
$ python3 list_bucket.py $s3_creds test-sync-bucket
obj-no-slash

3. Create object with slash called "object-/with-slash" on Primary

4. Object is not replicated.

From secondary:
$ python3 list_bucket.py $s3_creds test-sync-bucket
obj-no-slash

but the object is present on the Primary:
$ python3 list_bucket.py $s3_creds test-sync-bucket
obj-no-slash
object-/with-slash

Note: same happens if the object is created on the secondary, it will not be replicated on the primary

Expected result:
================

"object-/with-slash" is replicated correctly

[Giuseppe Petralia (peppepetra)]

Apparently adding to the apache2

AllowEncodedSlashes On

fix the issue.

[Trent Lloyd (lathiat)]

AllowEncodedSlashes seems to be recommended in a couple of places upstream so possibly a good fix?

https://linux.die.net/man/8/radosgw

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/html/ceph_object_gateway_for_ubuntu_x86_64/configure_ceph_object_gateway

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/842469

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/842469
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/8a844bca972ba40ba269b26111180732c0ab7312
Submitter: "Zuul (22348)"
Branch: master

commit 8a844bca972ba40ba269b26111180732c0ab7312
Author: peppepetra86 <email address hidden>
Date: Thu May 19 10:13:13 2022 +0200

    Allow URLs which contain encoded path separators

    This is to resolve the issue with objects containing slash in the name
    not correctly synced in multisite environments.

    Closes-Bug: #1974138
    Change-Id: I71ac000bb4754c9cb987d703f145dc2a5ff032ad

[Alan Baghumian (alanbach)]

I encountered this issue and adding "AllowEncodedSlashes On" on primary node's Apache worked and restarting it worked, but now I have a few 100GB of data being out of sync between primary and secondary buckets.

Tried following the steps here to force a sync but the actual sync does not happen:

https://www.ibm.com/docs/en/storage-ceph/7?topic=tmscog-synchronizing-data-in-multi-site-ceph-object-gateway-configuration

We need a real workaround to re-sync the data.

[Alan Baghumian (alanbach)]

Disabling and Enabling the Sync for a given bucket from the primary site does the re-sync trick:

$ sudo radosgw-admin bucket sync disable --bucket backup-archive
$ sudo radosgw-admin bucket sync enable --bucket backup-archive

$ sudo radosgw-admin bucket sync status --bucket backup-archive
          realm 0b1d60c6-5be8-470a-9d58-4e6a1215848b (hrizn.cloud)
      zonegroup 1be243bd-7041-4343-8202-3e47b2f28c76 (int.hrizn.cloud)
           zone 1d914e95-58bc-411c-a4fa-306dd2191344 (s3-az1)
         bucket :backup-archive[1d914e95-58bc-411c-a4fa-306dd2191344.22094132.2])

    source zone d46023eb-6350-4536-9440-7ea24ae270fa (s3-az2)
  source bucket :backup-archive[1d914e95-58bc-411c-a4fa-306dd2191344.22094132.2])
                full sync: 0/11 shards
                incremental sync: 11/11 shards
                bucket is behind on 1 shards
                behind shards: [8]

[Alan Baghumian (alanbach)]

Any ideas when a charm including this fix will be released?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/920114

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pacific
Review: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/920117

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/octopus
Review: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/920118

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/920117
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/64588c46ad5e3481d0c4e86b82afd4656b34726a
Submitter: "Zuul (22348)"
Branch: stable/pacific

commit 64588c46ad5e3481d0c4e86b82afd4656b34726a
Author: peppepetra86 <email address hidden>
Date: Thu May 19 10:13:13 2022 +0200

    Allow URLs which contain encoded path separators

    This is to resolve the issue with objects containing slash in the name
    not correctly synced in multisite environments.

    Closes-Bug: #1974138
    Change-Id: I71ac000bb4754c9cb987d703f145dc2a5ff032ad
    (cherry picked from commit 8a844bca972ba40ba269b26111180732c0ab7312)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/920114
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/f66e0742c4e3d795ff6bad393bbf22f1b3ede943
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit f66e0742c4e3d795ff6bad393bbf22f1b3ede943
Author: peppepetra86 <email address hidden>
Date: Thu May 19 10:13:13 2022 +0200

    Allow URLs which contain encoded path separators

    This is to resolve the issue with objects containing slash in the name
    not correctly synced in multisite environments.

    Closes-Bug: #1974138
    Change-Id: I71ac000bb4754c9cb987d703f145dc2a5ff032ad
    (cherry picked from commit 8a844bca972ba40ba269b26111180732c0ab7312)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ponnuvel Palaniyappan <email address hidden>" on branch: stable/octopus
Review: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/920118

[Alan Baghumian (alanbach)]

The udated charm was released. I just upgraded my Ceph Quincy backed RGW nodes and confirm the fix is now there. Thanks much!

[Kellen Renshaw (krenshaw)]

Has this been released onto stable/reef? I haven't been able to track down the answer yet.

[Dan Hill (hillpd)]

Yes this is fixed in reef. I double-checked and the fix is present in revision 574 sourced from the stable/reef channel.