swift3 middleware not working

[jani]

We installed Openstack newton version as per below URLS and swift is working fine without issues.

https://docs.openstack.org/newton/install-guide-rdo/keystone-users.html
https://docs.openstack.org/project-install-guide/object-storage/newton/get_started.html

[root@NFSSRV ~]# swift stat
                        Account: AUTH_59e1e54cc4cf4901a2a83fc1c733f701
                     Containers: 1
                        Objects: 1
                          Bytes: 267
Containers in policy "policy-0": 1
   Objects in policy "policy-0": 1
     Bytes in policy "policy-0": 267
    X-Account-Project-Domain-Id: default
                    X-Timestamp: 1496322906.93007
                     X-Trans-Id: tx38a35a8049024ac385622-00593157b4
                   Content-Type: text/plain; charset=utf-8
                  Accept-Ranges: bytes
[root@NFSSRV ~]#

later we configured swift3 middleware from below URL after which ran into below issues.
https://github.com/openstack/swift3

[root@NFSSRV ~]# systemctl restart openstack-swift-proxy.service
[root@NFSSRV ~]# systemctl status openstack-swift-proxy.service
Jun 02 11:47:44 NFSSRV swift-proxy-server[10821]: raise DistributionNotFound(req, requirers)
Jun 02 11:47:44 NFSSRV swift-proxy-server[10821]: pkg_resources.DistributionNotFound: The 'lxml' distribution was not found and is required by swi ft3

Then we installed lxml.
[root@NFSSRV ~]# yum install python-lxml
Installed:
  python-lxml.x86_64 0:3.2.1-4.el7

Proxy is not getting restarted again after lxml installation.

[root@NFSSRV ~]# systemctl restart openstack-swift-proxy.service
[root@NFSSRV ~]# systemctl status openstack-swift-proxy.service -l
? openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2017-06-02 11:55:23 IST; 6s ago
  Process: 11071 ExecStart=/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf (code=exited, status=1/FAILURE)
 Main PID: 11071 (code=exited, status=1/FAILURE)

Jun 02 11:55:23 NFSSRV swift-proxy-server[11071]: File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 640, in find_egg_entry_po int
Jun 02 11:55:23 NFSSRV swift-proxy-server[11071]: pkg_resources.require(self.spec)
Jun 02 11:55:23 NFSSRV swift-proxy-server[11071]: File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 654, in require
Jun 02 11:55:23 NFSSRV swift-proxy-server[11071]: needed = self.resolve(parse_requirements(requirements))
Jun 02 11:55:23 NFSSRV swift-proxy-server[11071]: File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 556, in resolve
Jun 02 11:55:23 NFSSRV swift-proxy-server[11071]: raise VersionConflict(dist,req) # XXX put more info here
Jun 02 11:55:23 NFSSRV swift-proxy-server[11071]: pkg_resources.VersionConflict: (swift 2.10.1 (/usr/lib/python2.7/site-packages), Requirement.par se('swift>=2.13.0'))
Jun 02 11:55:23 NFSSRV systemd[1]: openstack-swift-proxy.service: main process exited, code=exited, status=1/FAILURE
Jun 02 11:55:23 NFSSRV systemd[1]: Unit openstack-swift-proxy.service entered failed state.
Jun 02 11:55:23 NFSSRV systemd[1]: openstack-swift-proxy.service failed.

Please let us know what is the issue and how it can be resolved. This is a critical issue and quick reply will be really appreciated.

Thanks in advance. If need more info, will be happy to provide.

[Tim Burke (1-tim-z)]

Recently, swift3 started requiring a more-recent version of swift to be installed[1]. This allows the project to move forward (eventually) on various bugs[2][3], primarily related to multipart upload support, as well as new features such as bucket versioning support. I'd recommend upgrading to a more-recent version of swift (a lot of great stuff has landed since 2.10.1!) or, if staying on the Newton release is a requirement, installing an older version of swift3. The last release (1.11) can run with swift as old as 2.1.0; the last commit off of master that supported your version of swift was fc66da8.

[1] https://github.com/openstack/swift3/commit/31d4353
[2] https://bugs.launchpad.net/swift3/+bug/1522578 which needed at least https://github.com/openstack/swift/commit/2d25fe6 upstream
[3] https://bugs.launchpad.net/swift3/+bug/1636663 which needed https://github.com/openstack/swift/commit/bb71657 upstream

[jani (janibasha)]

Thanks so much for the update.

We tried below commits but are unable to configure the s3 and getting below errors.

Swift3 Tags:

1. swift3-31d4353bf19d00a786c9c3258a34c1189d1ef1a8

2. swift3-fc66da81a50b9362843029312ea9a6ae921c1e51

Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?
error in setup command: Error parsing /home/vedams/swift3-fc66da81a50b9362843029312ea9a6ae921c1e51/setup.cfg: Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?

Note:
Installed Git before Swift3 and after Swift3 setup still facing the same issue.

[jani (janibasha)]

We installed latest Oepnstack Ocata version and installed swift3 from master branch after which its working fine.

[root@Ocata ~]# swift list
dhanu
[root@Ocata ~]# openstack ec2 credentials create --user 47f69249ebda4fb38be4c7a3d7f97577 --project admin
+------------+------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------+
| access | 66adee966342469f9a4125b19da505e3 |
| links | {u'self': u'http://192.168.12.78:35357/v3/users/47f69249ebda4fb38be4c7a3d7f97577/credentials/OS- |
| | EC2/66adee966342469f9a4125b19da505e3'} |
| project_id | 489a40b011b74946b0e3e6a18f4e50c5 |
| secret | d2fa45b9e3be47bd964670d7d6c2b4e9 |
| trust_id | None |
| user_id | 47f69249ebda4fb38be4c7a3d7f97577 |
+------------+------------------------------------------------------------------------------------------------------------------------------------+
[root@Ocata ~]#

later if we try to send request from s3curl getting "403 Forbidden" error.

root@s3fsvm:~/s3curl# ./s3curl.pl --id="66adee966342469f9a4125b19da505e3" --key="d2fa45b9e3be47bd964670d7d6c2b4e9" http://192.168.12.78:8080 --debug

< HTTP/1.1 403 Forbidden
< x-amz-id-2: txf468d08f436a49c9a8149-005932a3fc
< x-amz-request-id: txf468d08f436a49c9a8149-005932a3fc
< Content-Type: application/xml
< X-Trans-Id: txf468d08f436a49c9a8149-005932a3fc
< X-Openstack-Request-Id: txf468d08f436a49c9a8149-005932a3fc
< Date: Sat, 03 Jun 2017 11:56:44 GMT
< Transfer-Encoding: chunked
<
<?xml version='1.0' encoding='UTF-8'?>
* Connection #0 to host 192.168.12.78 left intact
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>txf468d08f436a49c9a8149-005932a3fc</RequestId></Error>

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Could you share your proxy server config (in particular pipeline and swift3/authtoken/keystoneauth sections)?

And probably, the proxy-server logs when you got the 403 Forbidden will help us to find the problem.

Thanks

[jani (janibasha)]

I have attached the available proxy config file, keystone and swift logs.

Proxy and other logs are not available as they are not configured.

[Thiago da Silva (thiagodasilva)]

The pipeline in your proxy config is wrong, line 94 is like this:

pipeline = catch_errors gatekeeper healthcheck proxy-logging cache swift3 s3token container_sync bulk ratelimit authtoken keystoneauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

It should be:

pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken swift3 s3token keystoneauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

Can you please try that?

[Thiago da Silva (thiagodasilva)]

Also, it might be a good idea to join #openstack-swift on freenode to get some quick help, it might go faster then through the bug.

[Thiago da Silva (thiagodasilva)]

@jani, were you able to resolve this issue?