Failed to get object

Bug #1677095 reported by Feilong Wang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zaqar
New
Undecided
Unassigned

Bug Description

When using swift driver, sometimes I got below 401 error:

feilong@feilong-ThinkPad-X1-Carbon-2nd:~/MyWorkspace/workspace_community/zaqar$ openstack container list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils. This warning is caused by an out-of-date import in /usr/local/lib/python2.7/dist-packages/ironic_discoverd/shell.py
+--------------------------------------------------------+
| Name |
+--------------------------------------------------------+
| zaqar_message:myqueue:90a3964fd91848c4aa83b1ae42490acb |
| zaqar_queue:90a3964fd91848c4aa83b1ae42490acb |
+--------------------------------------------------------+
feilong@feilong-ThinkPad-X1-Carbon-2nd:~/MyWorkspace/workspace_community/zaqar$ openstack messages post myqueue '[{"body": "hello"}]' --os-profile 0xb613679a0814d9ec772f95d778c35fc5ff1697c493715653c6c712144292c5ad
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils. This warning is caused by an out-of-date import in /usr/local/lib/python2.7/dist-packages/ironic_discoverd/shell.py
Error response from Zaqar. Code: 500. Title: Internal server error. Description: Object GET failed: http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t.
Trace ID: 3ddc649a-815f-44c7-b371-f53aaab7d138
Display trace with command:
osprofiler trace show --html 3ddc649a-815f-44c7-b371-f53aaab7d138
feilong@feilong-ThinkPad-X1-Carbon-2nd:~/MyWorkspace/workspace_community/zaqar$ cat ~/keystonerc
# Use this for debugging issues before files in accrc are created

# Set up password auth credentials now that Keystone is bootstrapped
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://127.0.0.1/identity_admin
export OS_USERNAME=admin
export OS_USER_DOMAIN_ID=default
export OS_PASSWORD=passw0rd
export OS_PROJECT_NAME=demo
export OS_PROJECT_DOMAIN_ID=default
export OS_REGION_NAME=RegionOne

But as shown on above, the credentials are correct because the openstack client can list containers and nothing changed in zaqar.conf for swift driver.

[drivers:message_store:swift]
uri = swift://admin:passw0rd@/demo
auth_url = http://127.0.0.1:35357/v3

On zaqar server, I got below error:
2017-03-29 15:47:44.952 22495 DEBUG zaqar.common.decorators [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] Messages collection POST: {"project_id": "1f8650afd03e43d1addf47f28d52c8f0", "queue_name": "myqueue"} wrapper /home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/decorators.py:49
2017-03-29 15:47:44.982 22495 DEBUG zaqar.storage.pooling [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] No pool found for queue myqueue for project 1f8650afd03e43d1addf47f28d52c8f0 lookup /home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/storage/pooling.py:680
2017-03-29 15:47:44.987 22495 DEBUG zaqar.common.pipeline [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] Stage <zaqar.storage.swift.messages.MessageQueueHandler object at 0x7fd008665e90> does not implement get_metadata consumer /home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/pipeline.py:94
2017-03-29 15:47:45.078 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] REQ: curl -i http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue -X GET -H "X-Auth-Token: gAAAAABY2upVUAP-..."
2017-03-29 15:47:45.078 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] RESP STATUS: 401 Unauthorized
2017-03-29 15:47:45.079 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] RESP HEADERS: {u'Content-Length': u'131', u'Www-Authenticate': u'Swift realm="AUTH_1f8650afd03e43d1addf47f28d52c8f0", Keystone uri=\'http://127.0.0.1/identity\'', u'X-Trans-Id': u'tx6561bdae3e084a9ab20be-0058db2051', u'Date': u'Wed, 29 Mar 2017 02:47:45 GMT', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx6561bdae3e084a9ab20be-0058db2051'}
2017-03-29 15:47:45.079 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] RESP BODY: <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] Object GET failed: http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver Traceback (most recent call last):
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/falcon/api.py", line 209, in __call__
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver responder(req, resp, **params)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/decorators.py", line 50, in wrapper
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return func(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/transport/acl.py", line 41, in handler
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return func(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/transport/wsgi/v2_0/messages.py", line 167, in on_post
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver project_id)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/pipeline.py", line 97, in consumer
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver tmp = target(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return f(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/storage/pooling.py", line 256, in get_metadata
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return control.get_metadata(name, project=project)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/pipeline.py", line 97, in consumer
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver tmp = target(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/storage/swift/queues.py", line 74, in get_metadata
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver _, metadata = self._client.get_object(container, name)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1781, in get_object
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver headers=headers)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1673, in _retry
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver service_token=self.service_token, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1163, in get_object
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver raise ClientException.from_response(resp, 'Object GET failed', body)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver ClientException: Object GET failed: http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver
127.0.0.1 - - [29/Mar/2017 15:47:45] "POST /v2/queues/myqueue/messages HTTP/1.1" 500 307

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.