Cannot download an object, browser returns "HTTP/1.1 404 Not Found" error with a payload "NoSuchBucket"

Bug #1906499 reported by Przemyslaw Hausman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph RADOS Gateway Charm
New
Undecided
Unassigned

Bug Description

focal-ussuri, charms release 20.10
ceph-radosgw version 15.2.5, charm revision 291

It is impossible to download an object form a publicly visible container with a browser.
It is impossible to list a contents of a publicly visible container with a browser.

Both requests result with "HTTP/1.1 404 Not Found" error with a payload "NoSuchBucket".

ceph-radosgw is configured with config option namespace-tenants=true.

python3-swiftclient from focal-proposed is installed in openstack-dashboard units, as a workaround for bug "Cannot create a swift container, mandatory "Storage Policy" dropdown field is empty" https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1902944.

$ juju ssh openstack-dashboard/0 'apt policy python3-swiftclient'
python3-swiftclient:
  Installed: 1:3.9.0-0ubuntu1.1
  Candidate: 1:3.9.0-0ubuntu1.1
  Version table:
 *** 1:3.9.0-0ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:3.9.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

Swift (swift-storage charm) is not deployed in the environment.

Juju bundle: https://docs.google.com/document/d/1PHcAgrZZO8cOGl5sIZQLsH8z9AQqWS5QUoytvThvjMs/edit?usp=sharing

Logs
----

See attached ceph-client.rgw.txt for specific log entries from /var/log/ceph/ceph-client.rgw.*.log on ceph-radosgw unit.

Steps to reproduce
------------------

All actions are run as cloud admin (user: admin, project: admin, domain: admin_domain).

1. Create a container

$ CONTAINER=container-1
$ openstack container create ${CONTAINER} --max-width 72
+---------------------------+--------------+---------------------------+
| account | container | x-trans-id |
+---------------------------+--------------+---------------------------+
| AUTH_b4ee7e2e1a814aca9e06 | my-container | tx000000000000000000014-0 |
| a59f432c261c | | 05fc752c9-110c4-default |
+---------------------------+--------------+---------------------------+

$ openstack container show ${CONTAINER}
+--------------+---------------------------------------+
| Field | Value |
+--------------+---------------------------------------+
| account | AUTH_b4ee7e2e1a814aca9e06a59f432c261c |
| bytes_used | 0 |
| container | my-container |
| object_count | 0 |
+--------------+---------------------------------------+

2. Upload an object

$ echo "Hello!" > test.txt
$ openstack object create ${CONTAINER} test.txt
+----------+--------------+----------------------------------+
| object | container | etag |
+----------+--------------+----------------------------------+
| test.txt | my-container | e134ced312b3511d88943d57ccd70c83 |
+----------+--------------+----------------------------------+

3. Assign Access Control List to the container

Set 'read' ACL to make the container public:
$ swift post ${CONTAINER} --read-acl ".r:*,.rlistings"

Check updated ACLs on container, verify 'Read ACL'
$ swift stat ${CONTAINER}
                      Account: AUTH_b4ee7e2e1a814aca9e06a59f432c261c
                    Container: my-container
                      Objects: 1
                        Bytes: 7
                     Read ACL: .r:*,.rlistings
                    Write ACL:
                      Sync To:
                     Sync Key:
                       Server: Apache/2.4.41 (Ubuntu)
                  X-Timestamp: 1606898377.53094
X-Container-Bytes-Used-Actual: 4096
             X-Storage-Policy: default-placement
              X-Storage-Class: STANDARD
                Last-Modified: Wed, 02 Dec 2020 08:42:50 GMT
                   X-Trans-Id: tx00000000000000000001a-005fc75398-10910-default
       X-Openstack-Request-Id: tx00000000000000000001a-005fc75398-10910-default
                Accept-Ranges: bytes
                 Content-Type: text/plain; charset=utf-8

4. Generate URL for accessing the object in the browser

$ SWIFT=https://radosgw.example.com/swift/v1
$ ACCOUNT=$(openstack container show ${CONTAINER} -c account -f value)
$ echo "${SWIFT}/${ACCOUNT}/${CONTAINER}/test.txt"
https://radosgw.example.com/swift/v1/AUTH_b4ee7e2e1a814aca9e06a59f432c261c/my-container/test.txt

5. Attempt to download the object

$ curl -v https://radosgw.example.com/swift/v1/AUTH_b4ee7e2e1a814aca9e06a59f432c261c/my-container/test.txt
[...] Removed TLS handshake for brevity
> GET /swift/v1/AUTH_b4ee7e2e1a814aca9e06a59f432c261c/my-container/test.txt HTTP/1.1
> Host: radosgw.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 404 Not Found
< Date: Wed, 02 Dec 2020 08:47:28 GMT
< Server: Apache/2.4.41 (Ubuntu)
< Content-Length: 12
< X-Trans-Id: tx000000000000000000019-005fc754a0-108fe-default
< X-Openstack-Request-Id: tx000000000000000000019-005fc754a0-108fe-default
< Accept-Ranges: bytes
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host radosgw.example.com left intact

It is impossible to download the object.

Expected result
---------------

The publicly visible object can be downloaded with a browser or CLI utility (e.g. curl).

Additional notes
----------------

1. It is still possible to download the object with `openstack` CLI client:

$ openstack object save --file /tmp/saved-test.txt ${CONTAINER} test.txt
$ cat /tmp/saved-test.txt
Hello!

2. It is still possible to list objects in the container with CLI client:

$ openstack object list ${CONTAINER} --long --max-width 72
+----------+-------+----------------+--------------+-----------------+
| Name | Bytes | Hash | Content Type | Last Modified |
+----------+-------+----------------+--------------+-----------------+
| test.txt | 7 | e134ced312b351 | text/plain | 2020-12-02T08:4 |
| | | 1d88943d57ccd7 | | 1:46.914Z |
| | | 0c83 | | |
+----------+-------+----------------+--------------+-----------------+

Revision history for this message
Przemyslaw Hausman (phausman) wrote :
Revision history for this message
Przemyslaw Hausman (phausman) wrote :

This could be an upstream issue: "Broken public Swift bucket access with Keystone integration", https://tracker.ceph.com/issues/48382

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.