Cannot download an object, browser returns "HTTP/1.1 404 Not Found" error with a payload "NoSuchBucket"

[Przemyslaw Hausman]

focal-ussuri, charms release 20.10
ceph-radosgw version 15.2.5, charm revision 291

It is impossible to download an object form a publicly visible container with a browser.
It is impossible to list a contents of a publicly visible container with a browser.

Both requests result with "HTTP/1.1 404 Not Found" error with a payload "NoSuchBucket".

ceph-radosgw is configured with config option namespace-tenants=true.

python3-swiftclient from focal-proposed is installed in openstack-dashboard units, as a workaround for bug "Cannot create a swift container, mandatory "Storage Policy" dropdown field is empty" https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1902944.

$ juju ssh openstack-dashboard/0 'apt policy python3-swiftclient'
python3-swiftclient:
  Installed: 1:3.9.0-0ubuntu1.1
  Candidate: 1:3.9.0-0ubuntu1.1
  Version table:
 *** 1:3.9.0-0ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:3.9.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

Swift (swift-storage charm) is not deployed in the environment.

Juju bundle: https://docs.google.com/document/d/1PHcAgrZZO8cOGl5sIZQLsH8z9AQqWS5QUoytvThvjMs/edit?usp=sharing

Logs
----

See attached ceph-client.rgw.txt for specific log entries from /var/log/ceph/ceph-client.rgw.*.log on ceph-radosgw unit.

Steps to reproduce
------------------

All actions are run as cloud admin (user: admin, project: admin, domain: admin_domain).

1. Create a container

$ CONTAINER=container-1
$ openstack container create ${CONTAINER} --max-width 72
+---------------------------+--------------+---------------------------+
| account | container | x-trans-id |
+---------------------------+--------------+---------------------------+
| AUTH_b4ee7e2e1a814aca9e06 | my-container | tx000000000000000000014-0 |
| a59f432c261c | | 05fc752c9-110c4-default |
+---------------------------+--------------+---------------------------+

$ openstack container show ${CONTAINER}
+--------------+---------------------------------------+
| Field | Value |
+--------------+---------------------------------------+
| account | AUTH_b4ee7e2e1a814aca9e06a59f432c261c |
| bytes_used | 0 |
| container | my-container |
| object_count | 0 |
+--------------+---------------------------------------+

2. Upload an object

$ echo "Hello!" > test.txt
$ openstack object create ${CONTAINER} test.txt
+----------+--------------+----------------------------------+
| object | container | etag |
+----------+--------------+----------------------------------+
| test.txt | my-container | e134ced312b3511d88943d57ccd70c83 |
+----------+--------------+----------------------------------+

3. Assign Access Control List to the container

Set 'read' ACL to make the container public:
$ swift post ${CONTAINER} --read-acl ".r:*,.rlistings"

Check updated ACLs on container, verify 'Read ACL'
$ swift stat ${CONTAINER}
                      Account: AUTH_b4ee7e2e1a814aca9e06a59f432c261c
                    Container: my-container
                      Objects: 1
                        Bytes: 7
                     Read ACL: .r:*,.rlistings
                    Write ACL:
                      Sync To:
                     Sync Key:
                       Server: Apache/2.4.41 (Ubuntu)
                  X-Timestamp: 1606898377.53094
X-Container-Bytes-Used-Actual: 4096
             X-Storage-Policy: default-placement
              X-Storage-Class: STANDARD
                Last-Modified: Wed, 02 Dec 2020 08:42:50 GMT
                   X-Trans-Id: tx00000000000000000001a-005fc75398-10910-default
       X-Openstack-Request-Id: tx00000000000000000001a-005fc75398-10910-default
                Accept-Ranges: bytes
                 Content-Type: text/plain; charset=utf-8

4. Generate URL for accessing the object in the browser

$ SWIFT=https://radosgw.example.com/swift/v1
$ ACCOUNT=$(openstack container show ${CONTAINER} -c account -f value)
$ echo "${SWIFT}/${ACCOUNT}/${CONTAINER}/test.txt"
https://radosgw.example.com/swift/v1/AUTH_b4ee7e2e1a814aca9e06a59f432c261c/my-container/test.txt

5. Attempt to download the object

$ curl -v https://radosgw.example.com/swift/v1/AUTH_b4ee7e2e1a814aca9e06a59f432c261c/my-container/test.txt
[...] Removed TLS handshake for brevity
> GET /swift/v1/AUTH_b4ee7e2e1a814aca9e06a59f432c261c/my-container/test.txt HTTP/1.1
> Host: radosgw.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 404 Not Found
< Date: Wed, 02 Dec 2020 08:47:28 GMT
< Server: Apache/2.4.41 (Ubuntu)
< Content-Length: 12
< X-Trans-Id: tx000000000000000000019-005fc754a0-108fe-default
< X-Openstack-Request-Id: tx000000000000000000019-005fc754a0-108fe-default
< Accept-Ranges: bytes
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host radosgw.example.com left intact

It is impossible to download the object.

Expected result
---------------

The publicly visible object can be downloaded with a browser or CLI utility (e.g. curl).

Additional notes
----------------

1. It is still possible to download the object with `openstack` CLI client:

$ openstack object save --file /tmp/saved-test.txt ${CONTAINER} test.txt
$ cat /tmp/saved-test.txt
Hello!

2. It is still possible to list objects in the container with CLI client:

$ openstack object list ${CONTAINER} --long --max-width 72
+----------+-------+----------------+--------------+-----------------+
| Name | Bytes | Hash | Content Type | Last Modified |
+----------+-------+----------------+--------------+-----------------+
| test.txt | 7 | e134ced312b351 | text/plain | 2020-12-02T08:4 |
| | | 1d88943d57ccd7 | | 1:46.914Z |
| | | 0c83 | | |
+----------+-------+----------------+--------------+-----------------+

[Przemyslaw Hausman (phausman)]

This could be an upstream issue: "Broken public Swift bucket access with Keystone integration", https://tracker.ceph.com/issues/48382