Karbor

Unable to get Karbor endpoint from the operation engine

Bug #1798125 reported by Gaëtan Trellu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Karbor
New
Undecided
Unassigned

Bug Description

In karbor-operationengine.log I got an error because operation engine is not able to get the Karbor endpoint from Keystone catalog.

url.replace("$(", "%(") failed because url is equal to None.

This prevents me to use the trigger/scheduler.

==> /var/lib/docker/volumes/kolla_logs/_data/karbor/karbor-operationengine.log <==
2018-10-16 16:37:51.614 7 DEBUG karbor.services.operationengine.engine.triggers.timetrigger.time_trigger_multi_node [req-8aeb35b8-4a7b-4752-ad3d-f523932981bc - - - - -] Time trigger (1e8220f3-ba26-4071-ace4-1d909492de57) is due _loop /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/engine/triggers/timetrigger/time_trigger_multi_node.py:101
2018-10-16 16:37:51.615 7 DEBUG karbor.services.operationengine.engine.triggers.timetrigger.time_trigger_multi_node [req-8aeb35b8-4a7b-4752-ad3d-f523932981bc - - - - -] Rescheduling (1e8220f3-ba26-4071-ace4-1d909492de57) from 2018-10-16 16:37:49 to 2018-10-16 16:42:49 _loop /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/engine/triggers/timetrigger/time_trigger_multi_node.py:118
2018-10-16 16:37:52.112 7 DEBUG karbor.db.sqlalchemy.api [req-d380a92f-9a58-4be4-9f80-85309f72fdcc - - - - -] Updated trigger execution (0900292d-ac6f-4500-bc4a-849c7b290335) from 2018-10-16 16:37:49 to 2018-10-16 16:42:49 trigger_execution_update /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/db/sqlalchemy/api.py:502
2018-10-16 16:37:52.605 7 DEBUG karbor.services.operationengine.engine.triggers.timetrigger.time_trigger_multi_node [req-b8a7338a-ed1d-4631-9b3b-e3803dd1e119 - - - - -] Time trigger not yet due _loop /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/engine/triggers/timetrigger/time_trigger_multi_node.py:88
2018-10-16 16:37:53.389 7 DEBUG keystoneclient.session [req-e4d90796-533f-4a1c-b63e-a587b7f0ab8f - 2496480d09bc4c89acb4cb9ebc0101ce - - -] REQ: curl -g -i -X GET http://10.131.2.200:5000 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" _http_log_request /var/lib/kolla/venv/local/lib/python2.7/site-packages/keystoneclient/session.py:216
2018-10-16 16:37:53.395 7 DEBUG keystoneclient.session [req-e4d90796-533f-4a1c-b63e-a587b7f0ab8f - 2496480d09bc4c89acb4cb9ebc0101ce - - -] RESP: [300] Date: Tue, 16 Oct 2018 16:37:53 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token Content-Length: 598 Content-Type: application/json
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://10.131.2.200:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.131.2.200:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}
 _http_log_response /var/lib/kolla/venv/local/lib/python2.7/site-packages/keystoneclient/session.py:256
2018-10-16 16:37:53.782 7 DEBUG karbor.services.operationengine.karbor_client [req-e4d90796-533f-4a1c-b63e-a587b7f0ab8f - 2496480d09bc4c89acb4cb9ebc0101ce - - -] MSG010 None. get_karbor_endpoint /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/karbor_client.py:36
2018-10-16 16:37:53.784 7 DEBUG karbor.services.operationengine.karbor_client [req-e4d90796-533f-4a1c-b63e-a587b7f0ab8f - 2496480d09bc4c89acb4cb9ebc0101ce - - -] MSG011 <karbor.common.karbor_keystone_plugin.KarborKeystonePlugin object at 0x7f7bc9445650>. get_karbor_endpoint /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/karbor_client.py:37
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor [req-e4d90796-533f-4a1c-b63e-a587b7f0ab8f - 2496480d09bc4c89acb4cb9ebc0101ce - - -] Run operation(7220b09b-3e8d-43bc-886a-59c2d1c81be7) failed: AttributeError: 'NoneType' object has no attribute 'replace'
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor Traceback (most recent call last):
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/engine/executors/green_thread_executor.py", line 153, in _run_operation
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor param=param)
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operation_manager.py", line 57, in run_operation
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor op.run(operation_definition, **kwargs)
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/base.py", line 100, in run
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor self._execute(operation_definition, param)
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/protect_operation.py", line 46, in _execute
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor self._run(operation_definition, param, log_ref)
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/protect_operation.py", line 53, in _run
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor param.get("user_id"), param.get("project_id"))
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/base.py", line 199, in _create_karbor_client
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor karbor_url = self.karbor_endpoint % {"project_id": project_id}
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/base.py", line 64, in karbor_endpoint
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor self._karbor_endpoint = karbor_client.get_karbor_endpoint()
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/karbor_client.py", line 38, in get_karbor_endpoint
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor return url.replace("$(", "%(")
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor AttributeError: 'NoneType' object has no attribute 'replace'
2018-10-16 16:37:53.784 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor

See original description
Gaëtan Trellu (goldyfruit)
description: updated
description: updated
Revision history for this message
Gaëtan Trellu (goldyfruit) wrote :

Still getting the error, I though this change would help: https://review.openstack.org/#/c/616730/

Revision history for this message
Gaëtan Trellu (goldyfruit) wrote :
Download full text (4.1 KiB)

region_id option was missing in [karbor_client] section.
Now I got a different error:

2018-11-23 17:34:56.278 7 DEBUG karbor.common.karbor_keystone_plugin [req-3b788b56-b4d3-4c95-a98f-f21a5f465cee - 612bafb75fe044e0aca60db037bcf26e - - -] MSG001: [<Service description=Application Data Protection Service, enabled=True, id=9c4cf2135d4c45039e4ff88d81219509, links={u'self': u'http://10.128.150.100:35357/v3/services/9c4cf2135d4c45039e4ff88d81219509'}, name=karbor, type=data-protect>] get_service_endpoint /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/common/karbor_keystone_plugin.py:108
2018-11-23 17:34:56.279 7 DEBUG karbor.common.karbor_keystone_plugin [req-3b788b56-b4d3-4c95-a98f-f21a5f465cee - 612bafb75fe044e0aca60db037bcf26e - - -] MSG002: [<Endpoint enabled=True, id=5d5a80d606e3460b92d5be5638f35de0, interface=internal, links={u'self': u'http://10.128.150.100:35357/v3/endpoints/5d5a80d606e3460b92d5be5638f35de0'}, region=uat-ca-yul-1, region_id=uat-ca-yul-1, service_id=9c4cf2135d4c45039e4ff88d81219509, url=http://10.128.150.100:8799/v1/%(tenant_id)s>] get_service_endpoint /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/common/karbor_keystone_plugin.py:109
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor [req-3b788b56-b4d3-4c95-a98f-f21a5f465cee - 612bafb75fe044e0aca60db037bcf26e - - -] Run operation(0e6fd26e-5d84-4eda-bc3c-8877c261c41f) failed: KeyError: u'tenant_id'
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor Traceback (most recent call last):
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/engine/executors/green_thread_executor.py", line 153, in _run_operation
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor param=param)
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operation_manager.py", line 57, in run_operation
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor op.run(operation_definition, **kwargs)
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/base.py", line 100, in run
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor self._execute(operation_definition, param)
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/protect_operation.py", line 46, in _execute
2018-11-23 17:34:56.280 7 ERROR karbor.services.operationengine.engine.executors.green_thread_executor self._run(operation_definition, param, log_ref)
2018-11-23 17:34:5...

Read more...

Revision history for this message
Gaëtan Trellu (goldyfruit) wrote :

The error disappeared by itself :/
Now I don't have any more error but the checkpoint creation is not triggered, nothing happens.

2018-11-23 19:46:16.419 7 DEBUG karbor.services.operationengine.engine.triggers.timetrigger.time_trigger_multi_node [req-2ace0710-0837-456a-afb7-c01f23ab5995 - - - - -] Rescheduling (22777e7d-cbc8-4007-a41d-62a65fe54849) from 2018-11-23 19:46:14 to 2018-11-23 19:49:14 _loop /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/engine/triggers/timetrigger/time_trigger_multi_node.py:118
2018-11-23 19:46:16.425 7 DEBUG karbor.db.sqlalchemy.api [req-4973c7ba-a6ac-4a66-a813-0597a0c410cf - - - - -] Updated trigger execution (80bd22a0-5eea-4a04-b500-51b81ab415de) from 2018-11-23 19:46:14 to 2018-11-23 19:49:14 trigger_execution_update /var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/db/sqlalchemy/api.py:502

Revision history for this message
Gaëtan Trellu (goldyfruit) wrote :

The error came back after karbor-operationengine service restart.

2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor [req-3b067cb8-7a83-40b6-8116-d9f78d6efa82 - 612bafb75fe044e0aca60db037bcf26e - - -] Run operation(bc67cd3c-7987-4f5d-a874-85253d82378c) failed: KeyError: u'tenant_id'
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor Traceback (most recent call last):
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/engine/executors/green_thread_executor.py", line 153, in _run_operation
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor param=param)
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operation_manager.py", line 57, in run_operation
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor op.run(operation_definition, **kwargs)
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/base.py", line 100, in run
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor self._execute(operation_definition, param)
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/protect_operation.py", line 46, in _execute
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor self._run(operation_definition, param, log_ref)
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/protect_operation.py", line 53, in _run
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor param.get("user_id"), param.get("project_id"))
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/karbor/services/operationengine/operations/base.py", line 199, in _create_karbor_client
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor karbor_url = self.karbor_endpoint % {"project_id": project_id}
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor KeyError: u'tenant_id'
2018-11-28 18:13:17.209 8 ERROR karbor.services.operationengine.engine.executors.green_thread_executor

Revision history for this message
Gaëtan Trellu (goldyfruit) wrote :

Ok, I figured out why I got the "KeyError: u'tenant_id'" error, it is because Kolla creates the wrong endpoint. Karbor uses "project_id" and Kolla creates the endpoint using "tenant_id"

But still got nothing triggered, keep investigating !

Revision history for this message
Gaëtan Trellu (goldyfruit) wrote :

After some checks, the code never enter in the loop for operations_ids because operations_ids is equal to: set([])

https://github.com/openstack/karbor/blob/efc781783faf86bcecc6f2010ca822cc317643b5/karbor/services/operationengine/engine/triggers/timetrigger/time_trigger_multi_node.py#L213-L235

Revision history for this message
Gaëtan Trellu (goldyfruit) wrote :

By changing the "scheduling_strategy" from "multi-node" to "default" I'm able to get a checkpoint triggered!

Any idea why the "multi-node" strategy fails ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 8.0.0.0b1

This issue was fixed in the openstack/kolla-ansible 8.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 7.0.1

This issue was fixed in the openstack/kolla-ansible 7.0.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.