Memcached needs a restart after 'clean-up-token --purge-all'

Bug #1269444 reported by pushpesh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
swiftonfile
Fix Committed
Undecided
Unassigned

Bug Description

Steps to Reproduce:

# rm -rf /mnt/gluster-object/gsmetadata/*

# rm -rf /mnt/gluster-object/gsmetadata/.*
rm: cannot remove directory: `/mnt/gluster-object/gsmetadata/.'
rm: cannot remove directory: `/mnt/gluster-object/gsmetadata/..'

# gswauth-prep -K gswauthkey

# gswauth-add-account test -K gswauthkey

# gswauth-add-user -a test admin testing -K gswauthkey

# curl -X GET -H 'X-Auth-User: test:admin' -H 'X-Auth-Key: testing' http://localhost:8080/auth/v1.0 -v
* About to connect() to localhost port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8080
> Accept: */*
> X-Auth-User: test:admin
> X-Auth-Key: testing
>
< HTTP/1.1 200 OK
< Content-Length: 80
< X-Auth-Token-Expires: 86399
< X-Auth-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933
< X-Storage-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933
< X-Storage-Url: http://127.0.0.1:8080/v1/AUTH_test
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: txb825308e2a734ff3b1387-0052d5e77f
< Date: Wed, 15 Jan 2014 01:42:23 GMT
<
* Connection #0 to host localhost left intact
* Closing connection #0
{"storage": {"default": "local", "local": "http://127.0.0.1:8080/v1/AUTH_test"}}

# curl -X PUT -H 'X-Auth-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933' http://localhost:8080/v1/AUTH_test/dir11 -v
* About to connect() to localhost port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /v1/AUTH_test/dir11 HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8080
> Accept: */*
> X-Auth-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933
>
< HTTP/1.1 201 Created
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx2048995796dd4e65a1f11-0052d5e904
< Date: Wed, 15 Jan 2014 01:48:52 GMT
<
* Connection #0 to host localhost left intact
* Closing connection #0

# gswauth-cleanup-tokens -v --purge=test -K gswauthkeyGET test?marker=None
HEAD test/.services
HEAD test/admin
.token_3/AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933 purge account 'test'; deleting
DELETE .token_3/AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933
GET test?marker=admin
No more objects in test
Done.

# curl -X PUT -H 'X-Auth-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933' http://localhost:8080/v1/AUTH_test/dir12 -v
* About to connect() to localhost port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /v1/AUTH_test/dir12 HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8080
> Accept: */*
> X-Auth-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933
>
< HTTP/1.1 201 Created
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: txaa2683c6995b47e3b60b8-0052d5e94b
< Date: Wed, 15 Jan 2014 01:50:03 GMT
<
* Connection #0 to host localhost left intact
* Closing connection #0

# service memcached restart
Stopping memcached: [ OK ]
Starting memcached: [ OK ]

# curl -X PUT -H 'X-Auth-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933' http://localhost:8080/v1/AUTH_test/dir12 -v
* About to connect() to localhost port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /v1/AUTH_test/dir12 HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8080
> Accept: */*
> X-Auth-Token: AUTH_tkd4d2a5c8fdfe4cf9abd12f792e033933
>
< HTTP/1.1 401 Unauthorized
< Content-Length: 131
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: txcc7cc0c52a2c490b867d8-0052d5e959
< Date: Wed, 15 Jan 2014 01:50:17 GMT
<
* Connection #0 to host localhost left intact
* Closing connection #0
<html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>

Revision history for this message
Chetan Risbud (chetan-risbud) wrote :

Thanks Pushpesh for bringing this up. But it would be gr8 if we could have blueprint for this issue. IMO, this is an issue across all the authentication systems or perhaps any subsystem leveraged with memcache for performance reason. Some call back mechanism is required to make sure cache is flushed whenever call back is invoked.

Above said is just one of the approaches to solve the issue. blue print would help discuss other approaches.

Revision history for this message
Chetan Risbud (chetan-risbud) wrote :

Pushpesh, do you remember we changed gluster-swift-gen-builders to indicate "restart all swift services" as part of enhancement to that tool ? Could we add similar fix to this and move ahead while we figure out a real way of "memcached" invalidation ?

Revision history for this message
pushpesh (psharma) wrote :

Chetan ,

I agree with the above way to fix it 'temporarily '.

Revision history for this message
Chetan Risbud (chetan-risbud) wrote :
Changed in gluster-swift:
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.