Storlet app goes to 503 ServiceUnavaialble when the app doesn't set metadata

Bug #1652716 reported by Kota Tsuyuzaki
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
storlets
Confirmed
High
Unassigned

Bug Description

When a storlet app written in python which doesn't set user metadata in the __call__ method (i.e. it doesn't do like out_files[0].set_metadata({})), the app will fail with 503 ServiceUnavailable.

The actual response from swift client will be like:
Error downloading object 'storlet/test.py': Object GET failed: http://127.0.0.1:8080/v1/AUTH_107a9dd0b94547b5bb9c218c5609684a/storlet/test.py 503 Service Unavailable [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

And syslog said something like:

Dec 27 04:49:56 - object-server: Storlet execution failed: #012Traceback (most re
cent call last):#012 File "/usr/local/lib/python2.7/dist-packages/storlets/swift_middleware/storlet_handler.py", line 70, in __call__#012 return request_handler.handle_request()#012 File "/usr/local/lib/python2.7/dist-packages/storlets/swift_middleware/handlers/obj.py", line 70, in handle_request#012 return handler()#012 File "/usr/local/lib/python2.7/dist-packages/storlets/swift_middleware/handlers/obj.py", line 127, in GET#012 return self.apply_storlet(orig_resp)#012  File "/usr/local/lib/python2.7/dist-packages/storlets/swift_middleware/handlers/base.py", line 390, in apply_storlet#012 sresp = self._call_gateway(resp)#012 File "/usr/local/lib/python2.7/dist-packages/storlets/swift_middleware/handlers/obj.py", line 78, in _call_gateway#012 return self.gateway.invocation_flow(sreq)#012 File "/usr/local/lib/python2.7/dist-packages/storlets/gateway/gateways/docker/gateway.py", line 212, in invocation_flow#012 sresp = sprotocol.communicate()#012 File "/usr/local/lib/python2.7/dist-packages/storlets/gateway/gateways/docker/runtime.py", line 821, in communicate#012 out_md = self._read_metadata()#012 File "/usr/local/lib/python2.7/dist-packages/storlets/gateway/gateways/docker/runtime.py", line 772, in _read_metadata#0
12 raise StorletRuntimeException('Got invalid format about metadata')#012StorletRuntimeException: Got invalid format about metadata (txn: txd0daef8b5c434a8b9d6e4-005861f2f4)Dec 27 04:49:56 ip-10-184-153-157 proxy-server: ERROR 500 Storlet execution failed From Object Server 127.0.0.1:6613/sdb1 (txn: txd0daef8b5c434a8b9d6e4-005861f2f4)

For design specification, we need to decide to set empty dict or original metadata from the object when the app does nothing. I bet set original metadata may be useful.

description: updated
Revision history for this message
Eran Rom (eranr) wrote :

This is true for both Java and Python written storlets.
The initial thought behind this behaviour was to enforce the storlet writer to set the metadata.
I guess that a better behaviour would be:
1. Add a configuration flag: 'storlet_must_set_metadata'.
2. If the value is True then failing to set the metadata should result in an error.
3. If the value is False then upon timeout the middleware should probably set the original metadata of the storlet, while logging an info entry stating so: This can help for the case where the storlet indeed sets the metadata but failed to do so within the timeout.

Revision history for this message
Kota Tsuyuzaki (tsuyuzaki-kota) wrote :

Ok, Eran. That's expected.

If so, the temporary fix may be just changing the message from the 503 unavailable to 4xx (maybe bad request?) with message like 'Storlet app failed because it doesn't set metadata to the resposne' which is useful to let user know why the request failed, rather than "Error downloading object 'storlet/test.py': Object GET failed: http://127.0.0.1:8080/v1/AUTH_107a9dd0b94547b5bb9c218c5609684a/storlet/test.py 503 Service Unavailable [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently".

For design specification (it can be ok for future work), i think it would be nice to let the metadata handling can be by user trigger (i.e. sort of x-storlet-set-original-metadata: true).

summary: - python: storlet app goes to 503 ServiceUnavaialble when the app doesn't
- set metadata
+ Storlet app goes to 503 ServiceUnavaialble when the app doesn't set
+ metadata
Revision history for this message
Eran Rom (eranr) wrote :

The storlet not setting metadata isn't a problem with the request, but rather with the server so I think 5XX is more appropriate. Unrelated to the status we should definitely add a message as you have suggested.

Regarding the design specification remark. If we let the user control this, we can mask out buggy storlets that should change the metadata but don't (I admit that thinly example I have in mind is Content-Type which we currently do not allow to change anyway). Lets keep this in mind when we actually get to it.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.