| 2014-09-24 10:31:18 |
Feng Xi Yan |
bug |
|
|
added bug |
| 2014-09-24 10:40:32 |
Feng Xi Yan |
description |
Steps to reproduce:
(1) open nova api's audit, and restart nova-api
$> vim /etc/nova/api-paste.ini
[composite:openstack_compute_api_v2]
use = call:nova.api.auth:pipeline_factory
noauth = compute_req_id faultwrap sizelimit noauth ratelimit osapi_compute_app_v2
keystone = compute_req_id faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2
keystone_nolimit = compute_req_id faultwrap sizelimit authtoken keystonecontext audit osapi_compute_app_v2
[filter:audit]
paste.filter_factory = pycadf.middleware.audit:AuditMiddleware.factory
service_name = 10.104.0.160:8774
$> /etc/init.d/openstack-nova-api restart
(2) execute "nova list", will get a WARNING log in /var/log/ceilometer/collector.log
2014-09-24 17:33:20.253 2499 WARNING ceilometer.dispatcher.database [-] CL-53E95A5 message signature invalid, discarding message: {u'counter_name': u'http.response', u'user_id': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'resource_id': u'10.104.0.160:8774', u'timestamp': u'2014-09-24 09:33:19.784409', u'message_signature': u'c773872c48f095dcaed58882851cb36a3c3d7b13a151f2a9d8ffa795037e60ab', u'resource_metadata': {u'host': u'nova-api', u'request': {u'HTTP_X_TENANT_NAME': u'service', u'SCRIPT_NAME': u'/v2', u'REQUEST_METHOD': u'GET', u'PATH_INFO': u'/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_PROTOCOL': u'HTTP/1.0', u'HTTP_X_USER_ID': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'HTTP_USER_AGENT': u'python-novaclient', u'HTTP_X_DOMAIN_NAME': None, u'HTTP_X_PROJECT_DOMAIN_ID': u'default', u'REMOTE_PORT': u'56050', u'HTTP_X_ROLE': u'_member_,admin', u'HTTP_X_IDENTITY_STATUS': u'Confirmed', u'HTTP_X_SERVICE_NAME': u'10.104.0.160:8774', u'HTTP_X_DOMAIN_ID': None, u'SERVER_PORT': u'8774', u'CADF_EVENT': {u'typeURI': u'http://schemas.dmtf.org/cloud/audit/1.0/event', u'eventTime': u'2014-09-24T09:33:19.757466+0000', u'target': {u'typeURI': u'unknown', u'addresses': [{u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'admin'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'private'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'public'}], u'id': u'openstack:15be839cbda5454780d3f3f80ee7e4b2', u'name': u'nova'}, u'observer': {u'id': u'target'}, u'tags': [u'correlation_id?value=openstack:13070626-15dc-4281-b9e4-d2188a7d2922'], u'eventType': u'activity', u'initiator': {u'typeURI': u'service/security/account/user', u'name': u'ceilometer', u'credential': {u'token': u'MIIN2AYJKoZIhvcNAQcCoIINyTCCDcUC xxxxxxxx LArad+hi6COpZiM9XwCAbyrBTSawZJo=', u'identity_status': u'Confirmed'}, u'host': {u'agent': u'python-novaclient', u'address': u'10.104.0.160'}, u'project_id': u'openstack:bc88df306ada4f07b2664dc489686a15', u'id': u'openstack:3bc4b788378a4a8f9fd3a7f5ee2b7daf'}, u'reason': {u'reasonCode': u'200', u'reasonType': u'HTTP'}, u'reporterchain': [{u'reporterTime': u'2014-09-24T09:33:19.783754+0000', u'role': u'modifier', u'reporter': {u'id': u'target'}}], u'action': u'read', u'outcome': u'success', u'id': u'openstack:8bd9c6ed-2f60-4b3d-b24b-c74bf9c6fec8', u'requestPath': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f'}, u'HTTP_X_SERVICE_CATALOG': u'[{"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "id": "15be839cbda5454780d3f3f80ee7e4b2"}], "type": "compute", "name": "nova"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9696", "region": "RegionOne", "publicURL": "http://10.104.0.160:9696", "internalURL": "http://10.104.0.160:9696", "id": "3b1c00cf73a44325b241dcf69eb7ca5c"}], "type": "network", "name": "neutron"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9292", "region": "RegionOne", "publicURL": "http://10.104.0.160:9292", "internalURL": "http://10.104.0.160:9292", "id": "2b3977718eb84c91a8ec33943c206bcf"}], "type": "image", "name": "glance"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8777", "region": "RegionOne", "publicURL": "http://10.104.0.160:8777", "internalURL": "http://10.104.0.160:8777", "id": "06e40e1bf4534757a2b4cceed48fae85"}], "type": "metering", "name": "ceilometer"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8000/v1", "region": "RegionOne", "publicURL": "http://10.104.0.160:8000/v1", "internalURL": "http://10.104.0.160:8000/v1", "id": "72a2511306eb49b6b148b8f3556dc972"}], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "id": "372dfe6e6e574c17880e0a3823331d8f"}], "type": "volume", "name": "cinder"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "id": "07c225f1a8eb4a8d82e406507a8242a6"}], "type": "orchestration", "name": "heat"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:35357/v2.0", "region": "RegionOne", "publicURL": "http://10.104.0.160:5000/v2.0", "internalURL": "http://10.104.0.160:5000/v2.0", "id": "1faa00cc3ff4473ca72f96bc0eb08137"}], "type": "identity", "name": "keystone"}]', u'HTTP_X_TENANT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_PROJECT_DOMAIN_NAME': u'Default', u'HTTP_X_USER_DOMAIN_NAME': u'Default', u'HTTP_X_USER': u'ceilometer', u'HTTP_X_USER_DOMAIN_ID': u'default', u'HTTP_HOST': u'10.104.0.160:8774', u'HTTP_X_AUTH_PROJECT_ID': u'service', u'CADF_EVENT_CORRELATION_ID': u'openstack:13070626-15dc-4281-b9e4-d2188a7d2922', u'HTTP_X_TENANT': u'service', u'HTTP_ACCEPT': u'application/json', u'RAW_PATH_INFO': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_NAME': u'10.104.0.160', u'REMOTE_ADDR': u'10.104.0.160', u'HTTP_X_PROJECT_NAME': u'service', u'HTTP_X_PROJECT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_USER_NAME': u'ceilometer', u'CONTENT_TYPE': u'text/plain', u'GATEWAY_INTERFACE': u'CGI/1.1', u'HTTP_X_ROLES': u'_member_,admin', u'HTTP_ACCEPT_ENCODING': u'gzip, deflate'}, u'response': {u'status': u'200 OK', u'headers': {u'Content-Length': u'601', u'Content-Type': u'application/json'}}, u'event_type': u'http.response'}, u'source': u'openstack', u'counter_unit': u'response', u'counter_volume': 1, u'project_id': u'bc88df306ada4f07b2664dc489686a15', u'message_id': u'd2040b54-43cd-11e4-92be-fa163e2f929b', u'counter_type': u'delta'}
The reason of this issue is hidden so deeply:
When the audit samples are received by publisher, publisher will change the samples into metering messages, compute a signature, add the signature into message, and cast the message to qpid.
Then, the collector get the metering message from qpid, and validates the signature firstly. The error occurs in the signature validation process: the validation is failed because the signature generated is different from the signature before casting.
The reason that the signature validation fails is that the metering message is changed slightly after cased from qpid.
What is the change?
The message dict before casting has a item:
'reporterchain': [ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]
But after casted back from qpid, this item is changed into:
u'reporterchain': [ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]
The difference is that the item is encode into unicode.
When compute signature, this item will be transited into string format:
Before casting:
"[ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]"
After casting:
"[ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]"
So this slight difference result in different signature.
Note: This error only occurs on response message, but not request message, because request message has no the above item. |
Steps to reproduce:
(1) open nova api's audit, and restart nova-api
$> vim /etc/nova/api-paste.ini
[composite:openstack_compute_api_v2]
use = call:nova.api.auth:pipeline_factory
noauth = compute_req_id faultwrap sizelimit noauth ratelimit osapi_compute_app_v2
keystone = compute_req_id faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2
keystone_nolimit = compute_req_id faultwrap sizelimit authtoken keystonecontext audit osapi_compute_app_v2
[filter:audit]
paste.filter_factory = pycadf.middleware.audit:AuditMiddleware.factory
service_name = 10.104.0.160:8774
$> /etc/init.d/openstack-nova-api restart
(2) execute "nova list", will get a WARNING log in /var/log/ceilometer/collector.log
2014-09-24 17:33:20.253 2499 WARNING ceilometer.dispatcher.database [-] CL-53E95A5 message signature invalid, discarding message: {u'counter_name': u'http.response', u'user_id': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'resource_id': u'10.104.0.160:8774', u'timestamp': u'2014-09-24 09:33:19.784409', u'message_signature': u'c773872c48f095dcaed58882851cb36a3c3d7b13a151f2a9d8ffa795037e60ab', u'resource_metadata': {u'host': u'nova-api', u'request': {u'HTTP_X_TENANT_NAME': u'service', u'SCRIPT_NAME': u'/v2', u'REQUEST_METHOD': u'GET', u'PATH_INFO': u'/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_PROTOCOL': u'HTTP/1.0', u'HTTP_X_USER_ID': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'HTTP_USER_AGENT': u'python-novaclient', u'HTTP_X_DOMAIN_NAME': None, u'HTTP_X_PROJECT_DOMAIN_ID': u'default', u'REMOTE_PORT': u'56050', u'HTTP_X_ROLE': u'_member_,admin', u'HTTP_X_IDENTITY_STATUS': u'Confirmed', u'HTTP_X_SERVICE_NAME': u'10.104.0.160:8774', u'HTTP_X_DOMAIN_ID': None, u'SERVER_PORT': u'8774', u'CADF_EVENT': {u'typeURI': u'http://schemas.dmtf.org/cloud/audit/1.0/event', u'eventTime': u'2014-09-24T09:33:19.757466+0000', u'target': {u'typeURI': u'unknown', u'addresses': [{u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'admin'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'private'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'public'}], u'id': u'openstack:15be839cbda5454780d3f3f80ee7e4b2', u'name': u'nova'}, u'observer': {u'id': u'target'}, u'tags': [u'correlation_id?value=openstack:13070626-15dc-4281-b9e4-d2188a7d2922'], u'eventType': u'activity', u'initiator': {u'typeURI': u'service/security/account/user', u'name': u'ceilometer', u'credential': {u'token': u'MIIN2AYJKoZIhvcNAQcCoIINyTCCDcUC xxxxxxxx LArad+hi6COpZiM9XwCAbyrBTSawZJo=', u'identity_status': u'Confirmed'}, u'host': {u'agent': u'python-novaclient', u'address': u'10.104.0.160'}, u'project_id': u'openstack:bc88df306ada4f07b2664dc489686a15', u'id': u'openstack:3bc4b788378a4a8f9fd3a7f5ee2b7daf'}, u'reason': {u'reasonCode': u'200', u'reasonType': u'HTTP'}, u'reporterchain': [{u'reporterTime': u'2014-09-24T09:33:19.783754+0000', u'role': u'modifier', u'reporter': {u'id': u'target'}}], u'action': u'read', u'outcome': u'success', u'id': u'openstack:8bd9c6ed-2f60-4b3d-b24b-c74bf9c6fec8', u'requestPath': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f'}, u'HTTP_X_SERVICE_CATALOG': u'[{"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "id": "15be839cbda5454780d3f3f80ee7e4b2"}], "type": "compute", "name": "nova"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9696", "region": "RegionOne", "publicURL": "http://10.104.0.160:9696", "internalURL": "http://10.104.0.160:9696", "id": "3b1c00cf73a44325b241dcf69eb7ca5c"}], "type": "network", "name": "neutron"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9292", "region": "RegionOne", "publicURL": "http://10.104.0.160:9292", "internalURL": "http://10.104.0.160:9292", "id": "2b3977718eb84c91a8ec33943c206bcf"}], "type": "image", "name": "glance"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8777", "region": "RegionOne", "publicURL": "http://10.104.0.160:8777", "internalURL": "http://10.104.0.160:8777", "id": "06e40e1bf4534757a2b4cceed48fae85"}], "type": "metering", "name": "ceilometer"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8000/v1", "region": "RegionOne", "publicURL": "http://10.104.0.160:8000/v1", "internalURL": "http://10.104.0.160:8000/v1", "id": "72a2511306eb49b6b148b8f3556dc972"}], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "id": "372dfe6e6e574c17880e0a3823331d8f"}], "type": "volume", "name": "cinder"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "id": "07c225f1a8eb4a8d82e406507a8242a6"}], "type": "orchestration", "name": "heat"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:35357/v2.0", "region": "RegionOne", "publicURL": "http://10.104.0.160:5000/v2.0", "internalURL": "http://10.104.0.160:5000/v2.0", "id": "1faa00cc3ff4473ca72f96bc0eb08137"}], "type": "identity", "name": "keystone"}]', u'HTTP_X_TENANT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_PROJECT_DOMAIN_NAME': u'Default', u'HTTP_X_USER_DOMAIN_NAME': u'Default', u'HTTP_X_USER': u'ceilometer', u'HTTP_X_USER_DOMAIN_ID': u'default', u'HTTP_HOST': u'10.104.0.160:8774', u'HTTP_X_AUTH_PROJECT_ID': u'service', u'CADF_EVENT_CORRELATION_ID': u'openstack:13070626-15dc-4281-b9e4-d2188a7d2922', u'HTTP_X_TENANT': u'service', u'HTTP_ACCEPT': u'application/json', u'RAW_PATH_INFO': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_NAME': u'10.104.0.160', u'REMOTE_ADDR': u'10.104.0.160', u'HTTP_X_PROJECT_NAME': u'service', u'HTTP_X_PROJECT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_USER_NAME': u'ceilometer', u'CONTENT_TYPE': u'text/plain', u'GATEWAY_INTERFACE': u'CGI/1.1', u'HTTP_X_ROLES': u'_member_,admin', u'HTTP_ACCEPT_ENCODING': u'gzip, deflate'}, u'response': {u'status': u'200 OK', u'headers': {u'Content-Length': u'601', u'Content-Type': u'application/json'}}, u'event_type': u'http.response'}, u'source': u'openstack', u'counter_unit': u'response', u'counter_volume': 1, u'project_id': u'bc88df306ada4f07b2664dc489686a15', u'message_id': u'd2040b54-43cd-11e4-92be-fa163e2f929b', u'counter_type': u'delta'}
The reason of this issue is hidden so deeply:
When the audit samples are received by publisher, publisher will change the samples into metering messages, compute a signature, add the signature into message, and cast the message to qpid.
Then, the collector get the metering message from qpid, and validates the signature firstly. The error occurs in the signature validation process: the validation is failed because the signature generated is different from the signature before casting.
The reason that the signature validation fails is that the metering message is changed slightly after cased from qpid.
What is the change?
The message dict before casting has a item:
'reporterchain': [ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]
But after casted back from qpid, this item is changed into:
u'reporterchain': [ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]
The difference is that the item is encode into unicode. The unicode is added by json dump and loads when message is casted in qpid.
When compute signature, this item will be transited into string format:
Before casting:
"[ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]"
After casting:
"[ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]"
So this slight difference result in different signature.
Note: This error only occurs on response message, but not request message, because request message has no the above item. |
|
| 2014-09-24 10:43:43 |
Feng Xi Yan |
description |
Steps to reproduce:
(1) open nova api's audit, and restart nova-api
$> vim /etc/nova/api-paste.ini
[composite:openstack_compute_api_v2]
use = call:nova.api.auth:pipeline_factory
noauth = compute_req_id faultwrap sizelimit noauth ratelimit osapi_compute_app_v2
keystone = compute_req_id faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2
keystone_nolimit = compute_req_id faultwrap sizelimit authtoken keystonecontext audit osapi_compute_app_v2
[filter:audit]
paste.filter_factory = pycadf.middleware.audit:AuditMiddleware.factory
service_name = 10.104.0.160:8774
$> /etc/init.d/openstack-nova-api restart
(2) execute "nova list", will get a WARNING log in /var/log/ceilometer/collector.log
2014-09-24 17:33:20.253 2499 WARNING ceilometer.dispatcher.database [-] CL-53E95A5 message signature invalid, discarding message: {u'counter_name': u'http.response', u'user_id': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'resource_id': u'10.104.0.160:8774', u'timestamp': u'2014-09-24 09:33:19.784409', u'message_signature': u'c773872c48f095dcaed58882851cb36a3c3d7b13a151f2a9d8ffa795037e60ab', u'resource_metadata': {u'host': u'nova-api', u'request': {u'HTTP_X_TENANT_NAME': u'service', u'SCRIPT_NAME': u'/v2', u'REQUEST_METHOD': u'GET', u'PATH_INFO': u'/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_PROTOCOL': u'HTTP/1.0', u'HTTP_X_USER_ID': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'HTTP_USER_AGENT': u'python-novaclient', u'HTTP_X_DOMAIN_NAME': None, u'HTTP_X_PROJECT_DOMAIN_ID': u'default', u'REMOTE_PORT': u'56050', u'HTTP_X_ROLE': u'_member_,admin', u'HTTP_X_IDENTITY_STATUS': u'Confirmed', u'HTTP_X_SERVICE_NAME': u'10.104.0.160:8774', u'HTTP_X_DOMAIN_ID': None, u'SERVER_PORT': u'8774', u'CADF_EVENT': {u'typeURI': u'http://schemas.dmtf.org/cloud/audit/1.0/event', u'eventTime': u'2014-09-24T09:33:19.757466+0000', u'target': {u'typeURI': u'unknown', u'addresses': [{u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'admin'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'private'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'public'}], u'id': u'openstack:15be839cbda5454780d3f3f80ee7e4b2', u'name': u'nova'}, u'observer': {u'id': u'target'}, u'tags': [u'correlation_id?value=openstack:13070626-15dc-4281-b9e4-d2188a7d2922'], u'eventType': u'activity', u'initiator': {u'typeURI': u'service/security/account/user', u'name': u'ceilometer', u'credential': {u'token': u'MIIN2AYJKoZIhvcNAQcCoIINyTCCDcUC xxxxxxxx LArad+hi6COpZiM9XwCAbyrBTSawZJo=', u'identity_status': u'Confirmed'}, u'host': {u'agent': u'python-novaclient', u'address': u'10.104.0.160'}, u'project_id': u'openstack:bc88df306ada4f07b2664dc489686a15', u'id': u'openstack:3bc4b788378a4a8f9fd3a7f5ee2b7daf'}, u'reason': {u'reasonCode': u'200', u'reasonType': u'HTTP'}, u'reporterchain': [{u'reporterTime': u'2014-09-24T09:33:19.783754+0000', u'role': u'modifier', u'reporter': {u'id': u'target'}}], u'action': u'read', u'outcome': u'success', u'id': u'openstack:8bd9c6ed-2f60-4b3d-b24b-c74bf9c6fec8', u'requestPath': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f'}, u'HTTP_X_SERVICE_CATALOG': u'[{"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "id": "15be839cbda5454780d3f3f80ee7e4b2"}], "type": "compute", "name": "nova"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9696", "region": "RegionOne", "publicURL": "http://10.104.0.160:9696", "internalURL": "http://10.104.0.160:9696", "id": "3b1c00cf73a44325b241dcf69eb7ca5c"}], "type": "network", "name": "neutron"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9292", "region": "RegionOne", "publicURL": "http://10.104.0.160:9292", "internalURL": "http://10.104.0.160:9292", "id": "2b3977718eb84c91a8ec33943c206bcf"}], "type": "image", "name": "glance"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8777", "region": "RegionOne", "publicURL": "http://10.104.0.160:8777", "internalURL": "http://10.104.0.160:8777", "id": "06e40e1bf4534757a2b4cceed48fae85"}], "type": "metering", "name": "ceilometer"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8000/v1", "region": "RegionOne", "publicURL": "http://10.104.0.160:8000/v1", "internalURL": "http://10.104.0.160:8000/v1", "id": "72a2511306eb49b6b148b8f3556dc972"}], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "id": "372dfe6e6e574c17880e0a3823331d8f"}], "type": "volume", "name": "cinder"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "id": "07c225f1a8eb4a8d82e406507a8242a6"}], "type": "orchestration", "name": "heat"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:35357/v2.0", "region": "RegionOne", "publicURL": "http://10.104.0.160:5000/v2.0", "internalURL": "http://10.104.0.160:5000/v2.0", "id": "1faa00cc3ff4473ca72f96bc0eb08137"}], "type": "identity", "name": "keystone"}]', u'HTTP_X_TENANT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_PROJECT_DOMAIN_NAME': u'Default', u'HTTP_X_USER_DOMAIN_NAME': u'Default', u'HTTP_X_USER': u'ceilometer', u'HTTP_X_USER_DOMAIN_ID': u'default', u'HTTP_HOST': u'10.104.0.160:8774', u'HTTP_X_AUTH_PROJECT_ID': u'service', u'CADF_EVENT_CORRELATION_ID': u'openstack:13070626-15dc-4281-b9e4-d2188a7d2922', u'HTTP_X_TENANT': u'service', u'HTTP_ACCEPT': u'application/json', u'RAW_PATH_INFO': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_NAME': u'10.104.0.160', u'REMOTE_ADDR': u'10.104.0.160', u'HTTP_X_PROJECT_NAME': u'service', u'HTTP_X_PROJECT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_USER_NAME': u'ceilometer', u'CONTENT_TYPE': u'text/plain', u'GATEWAY_INTERFACE': u'CGI/1.1', u'HTTP_X_ROLES': u'_member_,admin', u'HTTP_ACCEPT_ENCODING': u'gzip, deflate'}, u'response': {u'status': u'200 OK', u'headers': {u'Content-Length': u'601', u'Content-Type': u'application/json'}}, u'event_type': u'http.response'}, u'source': u'openstack', u'counter_unit': u'response', u'counter_volume': 1, u'project_id': u'bc88df306ada4f07b2664dc489686a15', u'message_id': u'd2040b54-43cd-11e4-92be-fa163e2f929b', u'counter_type': u'delta'}
The reason of this issue is hidden so deeply:
When the audit samples are received by publisher, publisher will change the samples into metering messages, compute a signature, add the signature into message, and cast the message to qpid.
Then, the collector get the metering message from qpid, and validates the signature firstly. The error occurs in the signature validation process: the validation is failed because the signature generated is different from the signature before casting.
The reason that the signature validation fails is that the metering message is changed slightly after cased from qpid.
What is the change?
The message dict before casting has a item:
'reporterchain': [ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]
But after casted back from qpid, this item is changed into:
u'reporterchain': [ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]
The difference is that the item is encode into unicode. The unicode is added by json dump and loads when message is casted in qpid.
When compute signature, this item will be transited into string format:
Before casting:
"[ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]"
After casting:
"[ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]"
So this slight difference result in different signature.
Note: This error only occurs on response message, but not request message, because request message has no the above item. |
Steps to reproduce:
(1) open nova api's audit, and restart nova-api
$> vim /etc/nova/api-paste.ini
[composite:openstack_compute_api_v2]
use = call:nova.api.auth:pipeline_factory
noauth = compute_req_id faultwrap sizelimit noauth ratelimit osapi_compute_app_v2
keystone = compute_req_id faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2
keystone_nolimit = compute_req_id faultwrap sizelimit authtoken keystonecontext audit osapi_compute_app_v2
[filter:audit]
paste.filter_factory = pycadf.middleware.audit:AuditMiddleware.factory
service_name = 10.104.0.160:8774
$> /etc/init.d/openstack-nova-api restart
(2) execute "nova list", will get a WARNING log in /var/log/ceilometer/collector.log
2014-09-24 17:33:20.253 2499 WARNING ceilometer.dispatcher.database [-] CL-53E95A5 message signature invalid, discarding message: {u'counter_name': u'http.response', u'user_id': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'resource_id': u'10.104.0.160:8774', u'timestamp': u'2014-09-24 09:33:19.784409', u'message_signature': u'c773872c48f095dcaed58882851cb36a3c3d7b13a151f2a9d8ffa795037e60ab', u'resource_metadata': {u'host': u'nova-api', u'request': {u'HTTP_X_TENANT_NAME': u'service', u'SCRIPT_NAME': u'/v2', u'REQUEST_METHOD': u'GET', u'PATH_INFO': u'/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_PROTOCOL': u'HTTP/1.0', u'HTTP_X_USER_ID': u'3bc4b788378a4a8f9fd3a7f5ee2b7daf', u'HTTP_USER_AGENT': u'python-novaclient', u'HTTP_X_DOMAIN_NAME': None, u'HTTP_X_PROJECT_DOMAIN_ID': u'default', u'REMOTE_PORT': u'56050', u'HTTP_X_ROLE': u'_member_,admin', u'HTTP_X_IDENTITY_STATUS': u'Confirmed', u'HTTP_X_SERVICE_NAME': u'10.104.0.160:8774', u'HTTP_X_DOMAIN_ID': None, u'SERVER_PORT': u'8774', u'CADF_EVENT': {u'typeURI': u'http://schemas.dmtf.org/cloud/audit/1.0/event', u'eventTime': u'2014-09-24T09:33:19.757466+0000', u'target': {u'typeURI': u'unknown', u'addresses': [{u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'admin'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'private'}, {u'url': u'http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15', u'name': u'public'}], u'id': u'openstack:15be839cbda5454780d3f3f80ee7e4b2', u'name': u'nova'}, u'observer': {u'id': u'target'}, u'tags': [u'correlation_id?value=openstack:13070626-15dc-4281-b9e4-d2188a7d2922'], u'eventType': u'activity', u'initiator': {u'typeURI': u'service/security/account/user', u'name': u'ceilometer', u'credential': {u'token': u'MIIN2AYJKoZIhvcNAQcCoIINyTCCDcUC xxxxxxxx LArad+hi6COpZiM9XwCAbyrBTSawZJo=', u'identity_status': u'Confirmed'}, u'host': {u'agent': u'python-novaclient', u'address': u'10.104.0.160'}, u'project_id': u'openstack:bc88df306ada4f07b2664dc489686a15', u'id': u'openstack:3bc4b788378a4a8f9fd3a7f5ee2b7daf'}, u'reason': {u'reasonCode': u'200', u'reasonType': u'HTTP'}, u'reporterchain': [{u'reporterTime': u'2014-09-24T09:33:19.783754+0000', u'role': u'modifier', u'reporter': {u'id': u'target'}}], u'action': u'read', u'outcome': u'success', u'id': u'openstack:8bd9c6ed-2f60-4b3d-b24b-c74bf9c6fec8', u'requestPath': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f'}, u'HTTP_X_SERVICE_CATALOG': u'[{"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8774/v2/bc88df306ada4f07b2664dc489686a15", "id": "15be839cbda5454780d3f3f80ee7e4b2"}], "type": "compute", "name": "nova"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9696", "region": "RegionOne", "publicURL": "http://10.104.0.160:9696", "internalURL": "http://10.104.0.160:9696", "id": "3b1c00cf73a44325b241dcf69eb7ca5c"}], "type": "network", "name": "neutron"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:9292", "region": "RegionOne", "publicURL": "http://10.104.0.160:9292", "internalURL": "http://10.104.0.160:9292", "id": "2b3977718eb84c91a8ec33943c206bcf"}], "type": "image", "name": "glance"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8777", "region": "RegionOne", "publicURL": "http://10.104.0.160:8777", "internalURL": "http://10.104.0.160:8777", "id": "06e40e1bf4534757a2b4cceed48fae85"}], "type": "metering", "name": "ceilometer"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8000/v1", "region": "RegionOne", "publicURL": "http://10.104.0.160:8000/v1", "internalURL": "http://10.104.0.160:8000/v1", "id": "72a2511306eb49b6b148b8f3556dc972"}], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8776/v1/bc88df306ada4f07b2664dc489686a15", "id": "372dfe6e6e574c17880e0a3823331d8f"}], "type": "volume", "name": "cinder"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "region": "RegionOne", "publicURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "internalURL": "http://10.104.0.160:8004/v1/bc88df306ada4f07b2664dc489686a15", "id": "07c225f1a8eb4a8d82e406507a8242a6"}], "type": "orchestration", "name": "heat"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://10.104.0.160:35357/v2.0", "region": "RegionOne", "publicURL": "http://10.104.0.160:5000/v2.0", "internalURL": "http://10.104.0.160:5000/v2.0", "id": "1faa00cc3ff4473ca72f96bc0eb08137"}], "type": "identity", "name": "keystone"}]', u'HTTP_X_TENANT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_PROJECT_DOMAIN_NAME': u'Default', u'HTTP_X_USER_DOMAIN_NAME': u'Default', u'HTTP_X_USER': u'ceilometer', u'HTTP_X_USER_DOMAIN_ID': u'default', u'HTTP_HOST': u'10.104.0.160:8774', u'HTTP_X_AUTH_PROJECT_ID': u'service', u'CADF_EVENT_CORRELATION_ID': u'openstack:13070626-15dc-4281-b9e4-d2188a7d2922', u'HTTP_X_TENANT': u'service', u'HTTP_ACCEPT': u'application/json', u'RAW_PATH_INFO': u'/v2/bc88df306ada4f07b2664dc489686a15/flavors/0cf89011-5a36-41d9-a326-7b796234ea2f', u'SERVER_NAME': u'10.104.0.160', u'REMOTE_ADDR': u'10.104.0.160', u'HTTP_X_PROJECT_NAME': u'service', u'HTTP_X_PROJECT_ID': u'bc88df306ada4f07b2664dc489686a15', u'HTTP_X_USER_NAME': u'ceilometer', u'CONTENT_TYPE': u'text/plain', u'GATEWAY_INTERFACE': u'CGI/1.1', u'HTTP_X_ROLES': u'_member_,admin', u'HTTP_ACCEPT_ENCODING': u'gzip, deflate'}, u'response': {u'status': u'200 OK', u'headers': {u'Content-Length': u'601', u'Content-Type': u'application/json'}}, u'event_type': u'http.response'}, u'source': u'openstack', u'counter_unit': u'response', u'counter_volume': 1, u'project_id': u'bc88df306ada4f07b2664dc489686a15', u'message_id': u'd2040b54-43cd-11e4-92be-fa163e2f929b', u'counter_type': u'delta'}
The reason of this issue is hidden so deeply:
When the audit samples are received by publisher, publisher will change the samples into metering messages, compute a signature, add the signature into message, and cast the message to qpid.
Then, the collector get the metering message from qpid, and validates the signature firstly. The error occurs in the signature validation process: the validation is failed because the signature generated is different from the signature before casting.
The reason that the signature validation fails is that the metering message is changed slightly after cased from qpid.
What is the change?
The message dict before casting has a item:
'reporterchain': [ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]
But after casted back from qpid, this item is changed into:
u'reporterchain': [ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]
The difference is that the item is encode into unicode. The unicode is added by json dump and loads when message is casted in qpid.
When compute signature, this item will be transited into string format:
Before casting:
"[ {'reporterTime': '2014-09-04T06:05:51.844913+0000', 'role': 'modifier', 'reporter': { 'id': 'target' } } ]"
After casting:
"[ { u'reporterTime': u'2014-09-04T06:05:51.844913+0000', u'role': u'modifier', u'reporter': { u'id': u'target'}}]"
So this slight difference result in different signature.
Note: This error only occurs on http.response message, but not http.request message, because http.request message has no the above item. |
|
| 2014-09-24 10:49:09 |
OpenStack Infra |
ceilometer: status |
New |
In Progress |
|
| 2014-09-24 10:49:09 |
OpenStack Infra |
ceilometer: assignee |
|
Feng Xi Yan (yanfengxi) |
|
| 2014-10-08 08:30:37 |
Dina Belova |
ceilometer: importance |
Undecided |
Medium |
|
| 2014-10-08 08:30:47 |
Dina Belova |
ceilometer: milestone |
|
kilo-1 |
|
| 2014-10-10 13:08:03 |
OpenStack Infra |
ceilometer: status |
In Progress |
Fix Committed |
|
| 2014-12-02 11:01:25 |
Eoghan Glynn |
tags |
ceilometer qpid |
ceilometer juno-backport-potential qpid |
|
| 2014-12-02 11:02:01 |
Eoghan Glynn |
nominated for series |
|
ceilometer/juno |
|
| 2014-12-02 11:02:01 |
Eoghan Glynn |
bug task added |
|
ceilometer/juno |
|
| 2014-12-02 11:02:13 |
Eoghan Glynn |
ceilometer/juno: assignee |
|
Eoghan Glynn (eglynn) |
|
| 2014-12-02 11:02:17 |
Eoghan Glynn |
ceilometer/juno: importance |
Undecided |
Medium |
|
| 2014-12-02 11:02:22 |
Eoghan Glynn |
ceilometer/juno: status |
New |
In Progress |
|
| 2014-12-02 11:02:25 |
Eoghan Glynn |
ceilometer/juno: milestone |
|
2014.2.1 |
|
| 2014-12-03 13:48:46 |
Alan Pevec |
tags |
ceilometer juno-backport-potential qpid |
ceilometer qpid |
|
| 2014-12-03 15:57:31 |
OpenStack Infra |
ceilometer/juno: status |
In Progress |
Fix Committed |
|
| 2014-12-05 08:08:50 |
Alan Pevec |
ceilometer/juno: status |
Fix Committed |
Fix Released |
|
| 2014-12-18 15:57:13 |
Thierry Carrez |
ceilometer: status |
Fix Committed |
Fix Released |
|
| 2015-04-30 08:58:59 |
Thierry Carrez |
ceilometer: milestone |
kilo-1 |
2015.1.0 |
|
