Failed to get object

Bug #1677095 reported by Feilong Wang on 2017-03-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zaqar
Undecided
Unassigned

Bug Description

When using swift driver, sometimes I got below 401 error:

feilong@feilong-ThinkPad-X1-Carbon-2nd:~/MyWorkspace/workspace_community/zaqar$ openstack container list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils. This warning is caused by an out-of-date import in /usr/local/lib/python2.7/dist-packages/ironic_discoverd/shell.py
+--------------------------------------------------------+
| Name |
+--------------------------------------------------------+
| zaqar_message:myqueue:90a3964fd91848c4aa83b1ae42490acb |
| zaqar_queue:90a3964fd91848c4aa83b1ae42490acb |
+--------------------------------------------------------+
feilong@feilong-ThinkPad-X1-Carbon-2nd:~/MyWorkspace/workspace_community/zaqar$ openstack messages post myqueue '[{"body": "hello"}]' --os-profile 0xb613679a0814d9ec772f95d778c35fc5ff1697c493715653c6c712144292c5ad
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils. This warning is caused by an out-of-date import in /usr/local/lib/python2.7/dist-packages/ironic_discoverd/shell.py
Error response from Zaqar. Code: 500. Title: Internal server error. Description: Object GET failed: http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t.
Trace ID: 3ddc649a-815f-44c7-b371-f53aaab7d138
Display trace with command:
osprofiler trace show --html 3ddc649a-815f-44c7-b371-f53aaab7d138
feilong@feilong-ThinkPad-X1-Carbon-2nd:~/MyWorkspace/workspace_community/zaqar$ cat ~/keystonerc
# Use this for debugging issues before files in accrc are created

# Set up password auth credentials now that Keystone is bootstrapped
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://127.0.0.1/identity_admin
export OS_USERNAME=admin
export OS_USER_DOMAIN_ID=default
export OS_PASSWORD=passw0rd
export OS_PROJECT_NAME=demo
export OS_PROJECT_DOMAIN_ID=default
export OS_REGION_NAME=RegionOne

But as shown on above, the credentials are correct because the openstack client can list containers and nothing changed in zaqar.conf for swift driver.

[drivers:message_store:swift]
uri = swift://admin:passw0rd@/demo
auth_url = http://127.0.0.1:35357/v3

On zaqar server, I got below error:
2017-03-29 15:47:44.952 22495 DEBUG zaqar.common.decorators [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] Messages collection POST: {"project_id": "1f8650afd03e43d1addf47f28d52c8f0", "queue_name": "myqueue"} wrapper /home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/decorators.py:49
2017-03-29 15:47:44.982 22495 DEBUG zaqar.storage.pooling [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] No pool found for queue myqueue for project 1f8650afd03e43d1addf47f28d52c8f0 lookup /home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/storage/pooling.py:680
2017-03-29 15:47:44.987 22495 DEBUG zaqar.common.pipeline [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] Stage <zaqar.storage.swift.messages.MessageQueueHandler object at 0x7fd008665e90> does not implement get_metadata consumer /home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/pipeline.py:94
2017-03-29 15:47:45.078 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] REQ: curl -i http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue -X GET -H "X-Auth-Token: gAAAAABY2upVUAP-..."
2017-03-29 15:47:45.078 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] RESP STATUS: 401 Unauthorized
2017-03-29 15:47:45.079 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] RESP HEADERS: {u'Content-Length': u'131', u'Www-Authenticate': u'Swift realm="AUTH_1f8650afd03e43d1addf47f28d52c8f0", Keystone uri=\'http://127.0.0.1/identity\'', u'X-Trans-Id': u'tx6561bdae3e084a9ab20be-0058db2051', u'Date': u'Wed, 29 Mar 2017 02:47:45 GMT', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx6561bdae3e084a9ab20be-0058db2051'}
2017-03-29 15:47:45.079 22495 INFO swiftclient [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] RESP BODY: <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver [(None,) cbf281459cd0466b8ef8211dc79bccca 1f8650afd03e43d1addf47f28d52c8f0 - - -] [project_id:1f8650afd03e43d1addf47f28d52c8f0] Object GET failed: http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver Traceback (most recent call last):
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/falcon/api.py", line 209, in __call__
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver responder(req, resp, **params)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/decorators.py", line 50, in wrapper
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return func(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/transport/acl.py", line 41, in handler
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return func(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/transport/wsgi/v2_0/messages.py", line 167, in on_post
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver project_id)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/pipeline.py", line 97, in consumer
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver tmp = target(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return f(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/storage/pooling.py", line 256, in get_metadata
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver return control.get_metadata(name, project=project)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/common/pipeline.py", line 97, in consumer
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver tmp = target(*args, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/home/feilong/MyWorkspace/workspace_community/zaqar/zaqar/storage/swift/queues.py", line 74, in get_metadata
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver _, metadata = self._client.get_object(container, name)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1781, in get_object
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver headers=headers)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1673, in _retry
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver service_token=self.service_token, **kwargs)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1163, in get_object
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver raise ClientException.from_response(resp, 'Object GET failed', body)
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver ClientException: Object GET failed: http://127.0.0.1:8080/v1/AUTH_1f8650afd03e43d1addf47f28d52c8f0/zaqar_queue%3A1f8650afd03e43d1addf47f28d52c8f0/myqueue 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
2017-03-29 15:47:45.080 22495 ERROR zaqar.transport.wsgi.driver
127.0.0.1 - - [29/Mar/2017 15:47:45] "POST /v2/queues/myqueue/messages HTTP/1.1" 500 307

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers