internal error in symlink proxy

Bug #1806468 reported by Bulat Gaifullin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

I have enabled symlink middleware.

pipeline before:

#pipeline = catch_errors gatekeeper healthcheck cache container_sync
bulk tempurl ratelimit list-endpoints authtoken swift3 s3token
keystoneauth copy container-quotas account-quotas slo dlo
versioned_writes staticweb proxy-logging proxy-server

pipeline after:

pipeline = catch_errors gatekeeper healthcheck cache container_sync bulk
tempurl ratelimit authtoken s3api s3token keystoneauth list-endpoints
copy container-quotas account-quotas slo dlo versioned_writes staticweb
symlink proxy-logging proxy-server

after that request

http://127.0.0.1/v1/AUTH_f66c83966d824cf28c71a632e551bbe9/filename

returns internal error

#012Traceback (most recent call last):
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/catch_errors.py",
line 75, in handle_request
#012 resp = self._app_call(env)
#012 File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line
1196, in _app_call
#012 resp = self.app(env, self._start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/gatekeeper.py",
line 122, in __call__
#012 return self.app(env, gatekeeper_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/healthcheck.py",
line 52, in __call__
#012 return self.app(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/memcache.py",
line 109, in __call__
#012 return self.app(env, start_response)
#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line
1488, in _wsgify_self
#012 return func(self, Request(env))(env, start_response)
#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line
1488, in _wsgify_self
#012 return func(self, Request(env))(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/tempurl.py",
line 502, in __call__
#012 return self.app(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/ratelimit.py",
line 311, in __call__
#012 return self.app(env, start_response)
#012 File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in
__call__
#012 resp = self.call_func(req, *args, **kw)
#012 File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in
call_func
#012 return self.func(req, *args, **kwargs)
#012 File
"/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py",
line 335, in __call__
#012 response = req.get_response(self._app)
#012 File "/usr/lib/python2.7/site-packages/webob/request.py", line
1313, in send
#012 application, catch_exc_info=False)
#012 File "/usr/lib/python2.7/site-packages/webob/request.py", line
1277, in call_application
#012 app_iter = application(self.environ, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/s3api/s3api.py",
line 164, in __call__
#012 return resp(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/s3api/s3token.py",
line 214, in __call__
#012 return self._app(environ, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/keystoneauth.py",
line 234, in __call__
#012 return self.app(environ, keystone_start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/list_endpoints.py",
line 200, in __call__
#012 return self.app(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/copy.py", line
249, in __call__
#012 return self.app(env, start_response)
#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line
1488, in _wsgify_self
#012 return func(self, Request(env))(env, start_response)
#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line
1488, in _wsgify_self
#012 return func(self, Request(env))(env, start_response)swd
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/slo.py", line
1470, in __call__
#012 return self.handle_container_listing(req, start_response)swd
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/slo.py", line
1432, in handle_container_listing
#012 resp = req.get_response(self.app)
#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line
1080, in get_response
#012 status, headers, app_iter = self.call_application(application)
#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line
1064, in call_application
#012 app_iter = application(self.environ, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/dlo.py", line
405, in __call__
#012 return self.app(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/versioned_writes.py",
line 846, in __call__
#012 return self.app(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/staticweb.py",
line 537, in __call__
#012 return context.handle_container(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/staticweb.py",
line 387, in handle_container
#012 return self.app(env, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/symlink.py",
line 563, in __call__swd
#012 return context.handle_container(req, start_response)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/symlink.py",
line 284, in handle_container
#012 app_resp = self._process_json_resp(app_resp, req)
#012 File
"/usr/lib/python2.7/site-packages/swift/common/middleware/symlink.py",
line 298, in _process_json_resp
#012 body_json = json.loads(resp_body)
#012 File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
#012 return _default_decoder.decode(s)
#012 File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
#012 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
#012 File "/usr/lib64/python2.7/json/decoder.py", line 384, in
raw_decode
#012 raise ValueError("No JSON object could be decoded")
#012ValueError: No JSON object could be decoded (txn:
tx9e92f52c094d4b8994d84-005c024855) (client_ip: 127.0.0.1)

Revision history for this message
Tim Burke (1-tim-z) wrote :

Huh... this reminds me a lot of https://bugs.launchpad.net/swift/+bug/1758710 ... I wonder if the listing_formats middleware isn't getting inserted where we expected it to be inserted? Following https://review.openstack.org/#/c/449394/ the proxy should always be requesting JSON listings, and the listing_formats middleware *should* get placed far enough left in the pipeline that it middlewares like staticweb, dlo, and symlink will always get back JSON...

Revision history for this message
Tim Burke (1-tim-z) wrote :

I got a chance to dig into this a bit more, and I think the pipeline auto-insertion code might be getting confused by the fact that there's only one proxy-logging filter in the pipeline. We've expected two for a long while now (since Swift 1.8.0 or so [1]), would you mind adding a second proxy-logging between gatekeeper and healthcheck then trying again?

[1] https://github.com/openstack/swift/commit/a622349

Revision history for this message
Thiago da Silva (thiagodasilva) wrote :

@Bulat, were you able to solve the issue with Tim's suggestion or are you still seeing this issue?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.