proxy-server generates malformed URL on S3 request

[Volodymyr Litovka]

I'm trying to setup S3 test bench for developers, using Devstack (using stable/newton branch in devstack configuration file - local.conf). While I'm able to browse containers and objects using CLI (openstack container / object, swift), I can't get access to containers using s3curl. In logs (full log is attached) I see two different URLs at final stage:

------------- “openstack container list” command, issued locally
[ ... ]
proxy-server: Using identity: {'service_roles': [], 'roles': [u'admin'], 'project_domain': (u'default', u'Default'), 'auth_version': 3, 'user': (u'eac0298a83e44b12b2c08aa98e9b1c9a', u'admin'), 'user_domain': (u'default', u'Default'), 'tenant': (u'2d7365b17c8147e9aead99f870125d31', u'admin')} (txn: txda7984e9e1f04b7792920-005811ca49)
[ ... ]
proxy-server: de.vs.ta.ck de.vs.ta.ck 27/Oct/2016/09/35/05 GET /v1/AUTH_2d7365b17c8147e9aead99f870125d31%3Fformat%3Djson HTTP/1.0 200 - osc-lib%20keystoneauth1/2.14.0%20python-requests/2.11.1%20CPython/2.7.12 a5ef5769d7ef... - 42 - txda7984e9e1f04b7792920-005811ca49 - 0.0881 - - 1477560905.352745056 1477560905.440839052 -

You see correct URL in the request above.

------------- S3 session using s3curl from re.mo.te.host
[ ... ]
proxy-server: Using identity: {'service_roles': [], 'roles': [u'admin'], 'project_domain': (u'default', u'Default'), 'auth_version': 3, 'user': (u'eac0298a83e44b12b2c08aa98e9b1c9a', u'admin'), 'user_domain': (u'default', u'Default'), 'tenant': (u'2d7365b17c8147e9aead99f870125d31', u'admin')} (txn: tx61f057911f3e475eb1962-005811c95a)
[ ... ]
proxy-server: re.mo.te.host re.mo.te.host 27/Oct/2016/09/31/07 GET / HTTP/1.0 200 - curl/7.43.0 - - 219 - tx61f057911f3e475eb1962-005811c95a - 0.2074 - - 1477560666.966339111 1477560667.173743010 -

URL above is malformed and, of course, will return nothing. It seems something can bee wrong with proxy-server - having same information, it produces different request URLs for different kinds of access (swift client access vs remote S3 access).

For S3 access I created EC2 credentials:
/opt# openstack credential create --type ec2 --project admin admin '{"access" : "admin", "secret" : "adm1n0"}'
+------------+------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------+
| blob | {"access" : "admin", "secret" : "adm1n0"} |
| id | 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 |
| project_id | 2d7365b17c8147e9aead99f870125d31 |
| type | ec2 |
| user_id | eac0298a83e44b12b2c08aa98e9b1c9a |
+------------+------------------------------------------------------------------+

and, of course, there are created containers and objects for admin/admin:
/opt# openstack object list c0
+----------+
| Name |
+----------+
| list.txt |
+----------+

Any ideas on what's is wrong and how to go ahead? I'm ready to answer any questions and provide any additional information on this. Full log of proxy-server, as well proxy-server.conf are attached.

Thank you!

[Tim Burke (1-tim-z)]

I'm not sure I understand; the did the client get an error? It looks like everything was successful:

  GET / HTTP/1.0 200

I guess the trouble is that the proxy is logging the path that the user requested (in this case '/') rather than the backend path? You can have swift3 emit log lines for the individual subrequests using the force_swift_request_proxy_log config option [1]. Note that swift3 typically spawns several subrequests for any one client request.

[1] https://github.com/openstack/swift3/blob/1.11/etc/proxy-server.conf-sample#L110-L118

[Volodymyr Litovka (doka.ua)]

Hi Tim,

sorry, forgotten to add output of s3curl:

*# ./s3curl.pl --id stest -- http://s30-ams.yttm.work:8080/ | xmllint
--format -*
   % Total % Received % Xferd Average Speed Time Time Time
Current
                                  Dload Upload Total Spent Left Speed
100 219 100 219 0 0 272 0 --:--:-- --:--:--
--:--:-- 272
*<?xml version="1.0" encoding="UTF-8"?>**
**<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">**
** <Owner>**
** <ID>admin:admin</ID>**
** <DisplayName>admin:admin</DisplayName>**
** </Owner>**
** <Buckets/>**
**</ListAllMyBucketsResult>**
*

I tried also set 'allow_no_owner' and 's3_acl' to True, but with no
result, while there are containers and objects in object storage in
'admin' project for admin user:

# openstack container list
+------+
| Name |
+------+
| c0 |
+------+
# openstack object list c0
+----------+
| Name |
+----------+
| list.txt |
+----------+

Thank you!

On 10/27/16 7:04 PM, Tim Burke wrote:
> I'm not sure I understand; the did the client get an error? It looks
> like everything was successful:
>
> GET / HTTP/1.0 200
>
> I guess the trouble is that the proxy is logging the path that the user
> requested (in this case '/') rather than the backend path? You can have
> swift3 emit log lines for the individual subrequests using the
> force_swift_request_proxy_log config option [1]. Note that swift3
> typically spawns several subrequests for any one client request.
>
> [1] https://github.com/openstack/swift3/blob/1.11/etc/proxy-server.conf-
> sample#L110-L118
>

--
Volodymyr Litovka
   "Vision without Execution is Hallucination." -- Thomas Edison

[Volodymyr Litovka (doka.ua)]

After switching force_swift_request_proxy_log to True additional records appeared in proxy log:

proxy-server: 109.251.240.12 109.251.240.12 27/Oct/2016/20/33/47 GET /v1/AUTH_2d7365b17c8147e9aead99f870125d31%3Fformat%3Djson HTTP/1.0 200 - curl/7.43.0 23faf85bf6d0... - 42 - tx31edc32cea324e2fb0a8c-00581264aa - 0.0099 S3 - 1477600427.192111015 1477600427.202028036 -

which looks ok but nevertheless s3curl returns empty list. Probably, this is another problem in configuration, not related to initial message I wrote. May be, security issue, something like to ACLs?

Thank you.

[Tim Burke (1-tim-z)]

Ah! Try setting the dns_compliant_bucket_names option to False. By default, we use the naming requirements for non-US-East regions, so the name "c0" seems to be invalid; it looks like the bucket name is too short. From S3's docs,

> The rules for DNS-compliant bucket names are:
>
> * Bucket names must be at least 3 and no more than 63 characters long.
> * ...

See http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules

[Volodymyr Litovka (doka.ua)]

Hi Tim,

fantastic! Thanks a lot, this helped - now I'm able to create / delete
buckets/objects, put objects and copy them between buckets.

I highly appreciate you help, thanks again!

On 10/28/16 4:10 AM, Tim Burke wrote:
> Ah! Try setting the dns_compliant_bucket_names option to False. By
> default, we use the naming requirements for non-US-East regions, so the
> name "c0" seems to be invalid; it looks like the bucket name is too
> short. From S3's docs,
>
>> The rules for DNS-compliant bucket names are:
>>
>> * Bucket names must be at least 3 and no more than 63 characters long.
>> * ...
> See
> http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules
>

--
Volodymyr Litovka
   "Vision without Execution is Hallucination." -- Thomas Edison