RGW/Swift 404 error when listing/deleting a newly created empty bucket

[Narinder Gupta]

on openstack dashboard whenever we create a new container and list or delete the same container immediately then get the error object not found.

In logs, we are seeing a similar footprint as mentioned below.

https://www.spinics.net/lists/ceph-users/msg64642.html

[Camille Rodriguez (camille.rodriguez)]

I see the same issue. My setup is an openstack cloud focal/ussuri, ceph-radosgw is related to ceph-proxy in the same model (which is connected to an external ceph cluster). When I try to get the details or delete the container, here is the traceback from the ceph-radosgw unit, /var/log/ceph/ceph-client.rgw.juju-c037bf-0-lxd-3.log :

2021-05-18T13:50:29.007+0000 7fbe1affd700 1 ====== starting new request req=0x7fbe1aff65c0 =====
2021-05-18T13:50:29.007+0000 7fbe1affd700 1 op->ERRORHANDLER: err_no=-2000 new_err_no=-2000
2021-05-18T13:50:29.007+0000 7fbe1affd700 1 ====== req done req=0x7fbe1aff65c0 op status=0 http_status=400 latency=0s ======
2021-05-18T13:50:29.007+0000 7fbe1affd700 1 civetweb: 0x7fbf0063c010: 127.0.0.1 - - [18/May/2021:13:50:29 +0000] "GET /swift/v1/t/ HTTP/1.1" 400 327 - python-swiftclient-3.9.0
2021-05-18T13:50:29.227+0000 7fbe1affd700 1 ====== starting new request req=0x7fbe1aff65c0 =====
2021-05-18T13:50:29.231+0000 7fbe1affd700 1 ====== req done req=0x7fbe1aff65c0 op status=1902 http_status=204 latency=0.004000338s ======
2021-05-18T13:50:29.231+0000 7fbe1affd700 1 civetweb: 0x7fbf0063c010: 127.0.0.1 - - [18/May/2021:13:50:29 +0000] "GET /swift/v1/test/ HTTP/1.1" 204 520 - python-swiftclient-3.9.0
2021-05-18T13:50:34.107+0000 7fbe1affd700 1 ====== starting new request req=0x7fbe1aff65c0 =====
2021-05-18T13:50:34.107+0000 7fbe1affd700 1 ====== req done req=0x7fbe1aff65c0 op status=0 http_status=200 latency=0s ======
2021-05-18T13:50:34.107+0000 7fbe1affd700 1 civetweb: 0x7fbf0063c010: 127.0.0.1 - - [18/May/2021:13:50:34 +0000] "GET / HTTP/1.1" 200 439 - check_http/v2.2 (monitoring-plugins 2.2)
2021-05-18T13:50:38.400+0000 7fbe1affd700 1 ====== starting new request req=0x7fbe1aff65c0 =====
2021-05-18T13:50:38.404+0000 7fbe1affd700 1 op->ERRORHANDLER: err_no=-2002 new_err_no=-2002
2021-05-18T13:50:38.404+0000 7fbe1affd700 1 ====== req done req=0x7fbe1aff65c0 op status=0 http_status=404 latency=0.004000339s ======
2021-05-18T13:50:38.404+0000 7fbe1affd700 1 civetweb: 0x7fbf0063c010: 127.0.0.1 - - [18/May/2021:13:50:38 +0000] "HEAD /swift/v1/test2 HTTP/1.1" 404 308 - python-swiftclient-3.9.0
2021-05-18T13:50:43.444+0000 7fbe1affd700 1 ====== starting new request req=0x7fbe1aff65c0 =====
2021-05-18T13:50:43.444+0000 7fbe1affd700 1 op->ERRORHANDLER: err_no=-2002 new_err_no=-2002
2021-05-18T13:50:43.444+0000 7fbe1affd700 1 ====== req done req=0x7fbe1aff65c0 op status=0 http_status=404 latency=0s ======
2021-05-18T13:50:43.444+0000 7fbe1affd700 1 civetweb: 0x7fbf0063c010: 127.0.0.1 - - [18/May/2021:13:50:43 +0000] "GET /swift/v1/test2?format=json&limit=1001&delimiter=/ HTTP/1.1" 404 457 - python-swiftclient-3.9.0
2021-05-18T13:51:41.153+0000 7fbe1affd700 1 ====== starting new request req=0x7fbe1aff65c0 =====
2021-05-18T13:51:41.153+0000 7fbe1affd700 1 ====== req done req=0x7fbe1aff65c0 op status=0 http_status=200 latency=0s ======
2021-05-18T13:51:41.153+0000 7fbe1affd700 1 civetweb: 0x7fbf0063c010: 127.0.0.1 - - [18/May/2021:13:51:41 +0000] "GET / HTTP/1.1" 200 439 - check_http/v2.2 (monitoring-plugins 2.2)
2021-05-18T13:52:25.849+0000 7fbe1affd700 1 ====== starting new request req=0x7fbe1aff65c0 =====
2021...

[Camille Rodriguez (camille.rodriguez)]

Since the container isn't found, I'm also unable to create a folder inside of it.

[Camille Rodriguez (camille.rodriguez)]

deleting the container from the command line works

[Andre Ruiz (andre-ruiz)]

CLI seems to work ok, the problem is only in the dashboard. Some kind of out of sync cache. Also, if you let the containers sit for some time they eventually can be operated.

[Billy Olsen (billy-olsen)]

From the spinics post, the referenced bug https://tracker.ceph.com/issues/48632 was recently backported upstream to octopus. I can confirm that the version 15.2.8-0ubuntu0.20.04.1 included in focal-updates does not currently have this commit. I can also confirm that the 15.2.11-0ubuntu0.20.04.2 version in the focal-proposed pocket does.

I'd ask that you try upgrading the ceph packages on the radosgw units to the 15.2.11 version in the focal-proposed pocket to see if it resolves the problem.