SLO missing in pipeline because pipeline is confusing

[Archana]

Content Length of manifest file for large object is not equal to sum of content length of segments, instead its content length of manifest file.

Request

"PUT http://x.xx.xx.xxx:8091/v1/AUTH_f9d7c1cf6500469b80f0906f5f9b1791/arc19/testt?multipart-manifest=put"

# swift stat arc19 testt
       Account: AUTH_f9d7c1cf6500469b80f0906f5f9b1791
     Container: arc19
        Object: testt
  Content Type: application/octet-stream
Content Length: 303
 Last Modified: Wed, 05 Apr 2017 11:37:36 GMT
          ETag: 6bb12a356d8bad76fcae60beffb98c26
    Meta Mtime: 1491391761.014202
 Accept-Ranges: bytes
    Connection: keep-alive
   X-Timestamp: 1491392255.22155
    X-Trans-Id: txed26f76336ad40e0ba2e3-0058e4d70a
X-Static-Large-Object: true

[root@jupiter-vm564 ~]# swift list arc19_segments
testt/slo/1491391761.014202/36700160/33554432/00000000
testt/slo/1491391761.014202/36700160/33554432/00000001

Size of test5 file is 35MB
Content length of segments are 32MB and 3 MB
Content length of manifest id 303 bytes. 303 bytes is the length of manifest file. This is expected to be 35MB

[Janie Richling (jrichli)]

What version of both swift and python-swiftclient are you using? What is the body returned if you download testt - is it still the 35MB you expected to get? Can you reproduce this content-length error when using a curl GET or HEAD command to swift directly?

I am not able to reproduce this against the latest swift code (master). I tried using curl commands to swift directly, and using the latest python-swiftclient.

When I make a GET or HEAD request with the manifest path URL and the '?multipart-manifest=get' parameter, I correctly get the manifest body and the content-length of the manifest because of that parameter being sent with the request. And when I perform a GET or HEAD request without the parameter, I correctly get the body of the large object and the content-length of the large object returned.

[Archana (archupsg03)]

python-swiftclient 3.1.0

Swift Veersion 2.10

Body returned is the manifest file and not 35MB.

# swift download 142slo13 testt
testt [auth 0.418s, headers 0.554s, total 0.554s, 0.002 MB/s]
# ls -l testt
-rw-r--r-- 1 root root 309 Apr 5 09:55 testt

 curl -i http://x.xx.xx.xxx:8091/v1/AUTH_f9d7c1cf6500469b80f0906f5f9b1791/1423Apr6/sample?multipart-manifest=put -X GET -H "Content-Type: " -H "X-Auth-Token: 07d5709236524deeab45e63652e7e41a"
HTTP/1.1 200 OK
Content-Length: 311
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Thu, 06 Apr 2017 04:27:34 GMT
Etag: e7e1f14de9e622549dd51b000eec09a3
X-Timestamp: 1491452853.50530
X-Static-Large-Object: true
X-Object-Meta-Mtime: 1491452737.363976
X-Trans-Id: txc16173331665477283b12-0058e5c3fb
Date: Thu, 06 Apr 2017 04:28:43 GMT

[{"path": "/1423Apr6_segments/sample/slo/1491452737.363976/36700160/33554432/00000000", "etag": "58f06dd588d8ffb3beb46ada6309436b", "size_bytes": 33554432}, {"path": "/1423Apr6_segments/sample/slo/1491452737.363976/36700160/33554432/00000001", "etag": "d1dd210d6b1312cb342b56d02bd5e651", "size_bytes": 3145728}]

# ls -l sample
-rw-r--r-- 1 root root 35700160 Apr 6 00:25 sample

35MB got downloaded from the curl GET.

1. Manifest file content length is it expected to be length if the json for PUT operation

2. During GET operation, the json body is returned in the response and does the swift-client does the Content length sum ?

[Archana (archupsg03)]

# swift --debug download 1423Apr6 sample
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://x.xx.xx.x:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): x.xx.x.xx
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3898
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): x.xx.xx.xx
DEBUG:requests.packages.urllib3.connectionpool:"GET /v1/AUTH_f9d7c1cf6500469b80f0906f5f9b1791/1423Apr6/sample HTTP/1.1" 200 311
DEBUG:swiftclient:REQ: curl -i http://x.xx.xx.xxx:8091/v1/AUTH_f9d7c1cf6500469b80f0906f5f9b1791/1423Apr6/sample -X GET -H "X-Auth-Token: d48b7e179dc345a19c11b6294134b0f3"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'311', u'X-Object-Meta-Mtime': u'1491452737.363976', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Thu, 06 Apr 2017 04:27:34 GMT', u'Etag': u'e7e1f14de9e622549dd51b000eec09a3', u'X-Timestamp': u'1491452853.50530', u'X-Trans-Id': u'txf195f0fbbda1485ca03c9-0058e5cc1f', u'Date': u'Thu, 06 Apr 2017 05:03:27 GMT', u'X-Static-Large-Object': u'true', u'Content-Type': u'application/octet-stream'}
sample [auth 0.371s, headers 0.516s, total 0.516s, 0.002 MB/s]
# ls
sample
# ls -l sample
-rw-r--r-- 1 root root 311 Apr 6 00:25 sample

[clayg (clay-gerrard)]

this is fascinating - it's like the SLO middleware isn't in the pipeline:

ubuntu@saio:~$ swift download --debug test go1.8.linux-amd64.tar.gz
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): saio
DEBUG:requests.packages.urllib3.connectionpool:http://saio:8080 "GET /auth/v1.0 HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i http://saio:8080/auth/v1.0 -X GET
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'X-Trans-Id': u'tx678564ec862c48978c180-0058e5e3cc', u'X-Auth-Token-Expires': u'86348', u'X-Auth-Token': u'AUTH_tk6e65e18fadbd45b9922216445e4d8ac5', u'X-Storage-Token': u'AUTH_tk6e65e18fadbd45b9922216445e4d8ac5', u'Date': u'Thu, 06 Apr 2017 06:44:28 GMT', u'X-Storage-Url': u'http://saio:8080/v1/AUTH_test', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx678564ec862c48978c180-0058e5e3cc'}
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): saio
DEBUG:requests.packages.urllib3.connectionpool:http://saio:8080 "GET /v1/AUTH_test/test/go1.8.linux-amd64.tar.gz HTTP/1.1" 200 23489
DEBUG:swiftclient:REQ: curl -i http://saio:8080/v1/AUTH_test/test/go1.8.linux-amd64.tar.gz -X GET -H "X-Auth-Token: AUTH_tk6e65e18fadbd45b9922216445e4d8ac5"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'23489', u'Content-Type': u'application/x-tar', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Thu, 06 Apr 2017 06:43:50 GMT', u'Connection': u'close', u'Etag': u'21d29816a242e91aaa3d0958a954a7f2', u'X-Timestamp': u'1491461029.58568', u'X-Trans-Id': u'txa99c927da5e1471fa80e0-0058e5e3cc', u'Date': u'Thu, 06 Apr 2017 06:44:28 GMT', u'X-Static-Large-Object': u'True', u'X-Object-Meta-Mtime': u'1487278905.000000', u'X-Openstack-Request-Id': u'txa99c927da5e1471fa80e0-0058e5e3cc'}
go1.8.linux-amd64.tar.gz [auth 0.012s, headers 0.048s, total 0.055s, 0.538 MB/s]
ubuntu@saio:~$ ls -alhF go1.8.linux-amd64.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu 23K Feb 16 16:01 go1.8.linux-amd64.tar.gz

obvs, like Janie - when I put slo back in the pipeline by dlo and restart the proxy it "just works":

ubuntu@saio:~$ swift download --debug test go1.8.linux-amd64.tar.gz
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): saio
DEBUG:requests.packages.urllib3.connectionpool:http://saio:8080 "GET /auth/v1.0 HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i http://saio:8080/auth/v1.0 -X GET
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'X-Trans-Id': u'tx4bef61ee48814d6cb18d7-0058e5e425', u'X-Auth-Token-Expires': u'86259', u'X-Auth-Token': u'AUTH_tk6e65e18fadbd45b9922216445e4d8ac5', u'X-Storage-Token': u'AUTH_tk6e65e18fadbd45b9922216445e4d8ac5', u'Date': u'Thu, 06 Apr 2017 06:45:57 GMT', u'X-Storage-Url': u'http://saio:8080/v1/AUTH_test', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx4bef61ee48814d6cb18d7-0058e5e425'}
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): saio
DEBUG:requests.packages.urllib3.connectionpool:http://saio:8080 "GET /v1/AUTH_test/test/go1.8.linux-amd64.tar.gz HTTP/1.1" ...

[Archana (archupsg03)]

Is there any way to add SLO to pipeline ?

By default is it DLO ?

[Archana (archupsg03)]

proxy-server.conf

[DEFAULT]
bind_ip = x.x.x.x
bind_port = 8091
workers = 8
user = swift
[pipeline:main]
pipeline = healthcheck cache authtoken keystone proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
[filter:cache]
use = egg:swift#memcache
memcache_servers = 127.0.0.1:11211
[filter:catch_errors]
use = egg:swift#catch_errors
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:keystone]
use = egg:swift#keystoneauth
operator_roles = admin, SwiftOperator
is_admin = true
cache = swift.cache
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
admin_tenant_name = service
admin_user = swift
admin_password = passw0rd
auth_host = x.x.x.x
auth_port = 35357
auth_protocol = http
signing_dir = /tmp/keystone-signing-swift

swift/proxy/server.py

required_filters = [
    {'name': 'catch_errors'},
    {'name': 'gatekeeper',
     'after_fn': lambda pipe: (['catch_errors']
                               if pipe.startswith('catch_errors')
                               else [])},
    {'name': 'dlo', 'after_fn': lambda _junk: [
        'copy', 'staticweb', 'tempauth', 'keystoneauth',
        'catch_errors', 'gatekeeper', 'proxy_logging']},
    {'name': 'versioned_writes', 'after_fn': lambda _junk: [
        'slo', 'dlo', 'copy', 'staticweb', 'tempauth',
        'keystoneauth', 'catch_errors', 'gatekeeper', 'proxy_logging']},
    # Put copy before dlo, slo and versioned_writes
    {'name': 'copy', 'after_fn': lambda _junk: [
        'staticweb', 'tempauth', 'keystoneauth',
        'catch_errors', 'gatekeeper', 'proxy_logging']}]

proxy-server.conf doesn't have slo in its pipeline. The required filters are added into pipeline, which has dlo.

In SAIO documentation, I am seeing both slo and dlo included in the pipeline. In this case which will be triggered ? Is it based on the request ?

Is it good to have both slo and dlo in pipeline ?

[clayg (clay-gerrard)]

oh wow! You can set X-Static-Large-Object on PUT even if SLO is not in the pipeline!?

OH WOW! but then you can't *download* them once you enable SLO!?

Apr 6 03:53:22 saio proxy-server: Error: An error occurred: #012Traceback (most recent call last):#012 File "/vagrant/swift/swift/common/middleware/catch_errors.py", line 41, in handle_request#012 resp = self._app_call(env)#012 File "/vagrant/swift/swift/common/wsgi.py", line 1058, in _app_call#012 resp = self.app(env, self._start_response)#012 File "/vagrant/swift/swift/common/middleware/gatekeeper.py", line 102, in __call__#012 return self.app(env, gatekeeper_response)#012 File "/vagrant/swift/swift/common/middleware/healthcheck.py", line 52, in __call__#012 return self.app(env, start_response)#012 File "/vagrant/swift/swift/common/middleware/proxy_logging.py", line 346, in __call__#012 six.reraise(exc_type, exc_value, exc_traceback)#012 File "/vagrant/swift/swift/common/middleware/proxy_logging.py", line 338, in __call__#012 iterable = self.app(env, my_start_response)#012 File "/vagrant/swift/swift/common/middleware/memcache.py", line 109, in __call__#012 return self.app(env, start_response)#012 File "/vagrant/swift/swift/common/swob.py", line 1402, in _wsgify_self#012 return func(self, Request(env))(env, start_response)#012 File "/vagrant/swift/swift/common/swob.py", line 1402, in _wsgify_self#012 return func(self, Request(env))(env, start_response)#012 File "/vagrant/swift/swift/common/middleware/tempurl.py", line 401, in __call__#012 return self.app(env, start_response)#012 File "/vagrant/swift/swift/common/middleware/tempauth.py", line 292, in __call__#012 return self.app(env, start_response)#012 File "/vagrant/swift/swift/common/middleware/staticweb.py", line 531, in __call__#012 return self.app(env, start_response)#012 File "/vagrant/swift/swift/common/middleware/copy.py", line 344, in __call__#012 return self.app(env, start_response)#012 File "/vagrant/swift/swift/common/middleware/slo.py", line 1189, in __call__#012 return self.handle_multipart_get_or_head(req, start_response)#012 File "/vagrant/swift/swift/common/middleware/slo.py", line 904, in handle_multipart_get_or_head#012 return SloGetContext(self).handle_slo_get_or_head(req, start_response)#012 File "/vagrant/swift/swift/common/middleware/slo.py", line 705, in handle_slo_get_or_head#012 req, resp_headers, resp_iter)#012 File "/vagrant/swift/swift/common/middleware/slo.py", line 775, in get_or_head_response#012 etag.update(seg_dict['hash'])#012KeyError: 'hash' (txn: txb774a458007e46a988cd4-0058e5f3f2)

such a cool failure mode.

This is why we use sysmeta people! Let us learn from our past mistakes. ;)

@Archana yeah dlo (being older i guess) managed to get some auto-insertion support:

Apr 6 03:59:31 saio proxy-server: Adding required filter dlo to pipeline at position 10

Grrr.. "pipeline" doens't appear anywhere on https://docs.openstack.org/developer/swift/overview_large_objects.html#module-swift.common.middleware.slo - maybe this bug is half that - "you HAVE to turn on SLO in the pipeline". I hear people hate featureization because of interop - it...

[clayg (clay-gerrard)]

pipeline = healthcheck cache authtoken keystone proxy-server

^ that is a *very* minimal pipeline, one from devstack looks like this:

pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit crossdomain authtoken keystoneauth tempauth formpost staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

one one my dev environment (or at least the one I have handy) looks like this (currently):

pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl tempauth staticweb copy dlo versioned_writes keymaster encryption proxy-logging proxy-server

random lab server looks like this:

pipeline = catch_errors gatekeeper healthcheck proxy-logging cache formpost tempurl swc swiftstack_authen swiftstack_authz copy dlo slo proxy-logging proxy-server

swc swiftstack_authen swiftstack_authz are all bs custom stuff that do similar stuff to staticweb and keystone filters

Wow... you know I sorta forgot cause we haven't talked about it in a while. But I *hate* pipelines?

[Archana (archupsg03)]

Adding SLO to the pipeline worked.

So does, it mean we need to have both the middleware in the pipeline and based on the request the appropriate methods will be triggered ?

[Mahati Chamarthy (mahati-chamarthy)]

Archana, it is triggered by the headers you pass for slo.. In case you missed it, this reference link has description with examples of how dlo slo differ and how they work: https://docs.openstack.org/developer/swift/overview_large_objects.html

[Mahati Chamarthy (mahati-chamarthy)]

Like what Clay says - I think it makes sense to send a warning when X-Static-Large-Object is used on PUT and there is no SLO in pipeline. Not sure of the implications of inserting it automatically from a certain version though. On the face of it, it seems not an issue.

[Archana (archupsg03)]

For SLO no header is required is what I understand from the doc. X-Static-Large-Object is internal header.

For DLO we will be adding X-Object-manifest in the header

[Mahati Chamarthy (mahati-chamarthy)]

Archana - correct, I intended to refer to the parameters like ?multipart-manifest=put

[Archana (archupsg03)]

I feel this as a minor bug. Swift configuration should add both the middlewares in pipeline by default

[Archana (archupsg03)]

Does etag computation for DLO and SLO differs ?

DLO - Etag is MD5 of the file

SLO - Etag is MD5 of concatenated etag of the segment

[Archana (archupsg03)]

DLO - MD5 of the manifest file

[clayg (clay-gerrard)]

@Archana thank you for reporting this bug.

More information about Large Object Etags are documented here:

https://docs.openstack.org/developer/swift/overview_large_objects.html#additional-notes

If discover an additional issue please open a *different* bug. If you have questions or would like more information related to Large Object support please ask on one of the OpenStack ML [1] - or come join us on IRC in #openstack-swift on Freenode.

1. https://wiki.openstack.org/wiki/Mailing_Lists

[clayg (clay-gerrard)]

N.B. swiftclient will no longer send the (essentially private sysmeta) x-static-large-object header https://review.openstack.org/#/c/455470/

But it will still upload an object with ?multipart-manifest=put even when SLO isn't enabled.

[Thiago da Silva (thiagodasilva)]

I'm confused what this bug is about, or what do we intend to fix here? Are we proposing that SLO should be automatically added to the pipeline like DLO is or something else?