watcher

Compute CDM failed to be created under multiple cells environment

Bug #1709544 reported by Hidekazu Nakamura
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
watcher
Fix Released
Undecided
Unassigned

Bug Description

Under Multiple Cells environment, Compute CDM failed to be created with NoUniqueMatch exception.

Decision-engine log is the following:

 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 ERROR watcher.decision_engine.scheduling [req-7cd5f093-3c20-4afb-8e15-549a3d7d38ed None None] : NoUniqueMatch
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling Traceback (most recent call last):
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/scheduling.py", line 74, in _sync
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling timed_sync()
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/scheduling.py", line 62, in _timed_s
ync
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling sync_func()
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/base.py", line 185,
in synchronize
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling self.cluster_data_model = self.execute()
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 63, i
n execute
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling return builder.execute()
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 368, in execute
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling self._add_physical_layer()
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 99, in _add_physical_layer
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling self.add_compute_node(cnode)
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 103, in add_compute_node
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling compute_node = self.build_compute_node(node)
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 132, in build_compute_node
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling compute_service = self.nova_helper.get_service(node.service["id"])
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/common/nova_helper.py", line 71, in get_service
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling return self.nova.services.find(id=service_id)
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 418, in find
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling raise exceptions.NoUniqueMatch
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling NoUniqueMatch
 8月 09 14:51:43 w001 watcher-decision-engine[18864]: 2017-08-09 14:51:43.849 TRACE watcher.decision_engine.scheduling

services table is the following:

mysql> use nova_cell1;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from services;
+---------------------+---------------------+------------+----+------+----------------+-----------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+
| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid |
+---------------------+---------------------+------------+----+------+----------------+-----------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+
| 2017-08-09 05:16:25 | 2017-08-09 06:27:51 | NULL | 1 | w001 | nova-conductor | conductor | 429 | 0 | 0 | NULL | 2017-08-09 06:27:51 | 0 | 21 | 9f8ba5ba-9220-42ab-9ba8-771f64fb6d9b |
| 2017-08-09 05:19:46 | 2017-08-09 06:27:52 | NULL | 3 | w012 | nova-compute | compute | 409 | 0 | 0 | NULL | 2017-08-09 06:27:52 | 0 | 21 | d92ee6cd-6376-494c-ad15-b59c13dd4f21 |
| 2017-08-09 05:32:19 | 2017-08-09 06:27:55 | NULL | 4 | w013 | nova-compute | compute | 334 | 0 | 0 | NULL | 2017-08-09 06:27:55 | 0 | 21 | 7a570d78-7f38-4917-a7f4-b2c659cb7858 |
+---------------------+---------------------+------------+----+------+----------------+-----------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+
3 rows in set (0.00 sec)

mysql> use nova_cell0;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from services;
+---------------------+---------------------+------------+----+------+--------------------+-------------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+
| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid |
+---------------------+---------------------+------------+----+------+--------------------+-------------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+
| 2017-08-09 05:14:56 | NULL | NULL | 1 | w001 | nova-osapi_compute | NULL | 0 | 0 | 0 | NULL | NULL | 0 | 21 | 63d1bb03-e19d-4ec4-ba02-643093fa7ec7 |
| 2017-08-09 05:16:15 | 2017-08-09 06:28:32 | NULL | 3 | w001 | nova-scheduler | scheduler | 434 | 0 | 0 | NULL | 2017-08-09 06:28:32 | 0 | 21 | 11091b1b-3eac-4808-ab92-15b5c9f8e7b9 |
| 2017-08-09 05:16:22 | 2017-08-09 06:28:39 | NULL | 4 | w001 | nova-consoleauth | consoleauth | 434 | 0 | 0 | NULL | 2017-08-09 06:28:39 | 0 | 21 | a71d3e28-3d50-479e-a494-a60db3f59fdb |
| 2017-08-09 05:16:24 | NULL | NULL | 5 | w001 | nova-metadata | NULL | 0 | 0 | 0 | NULL | NULL | 0 | 21 | 14639e0e-7f51-4e69-859a-8550edd2b8e3 |
| 2017-08-09 05:16:25 | 2017-08-09 06:28:41 | NULL | 6 | w001 | nova-conductor | conductor | 434 | 0 | 0 | NULL | 2017-08-09 06:28:41 | 0 | 21 | d88b14b2-765e-453c-8191-479e4ae7a01b |
+---------------------+---------------------+------------+----+------+--------------------+-------------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+
5 rows in set (0.00 sec)

Hidekazu Nakamura (nakamura-h)
summary: - Compute CDM failed to be created
+ Compute CDM failed to be created under multiple cells environment
Revision history for this message
Hidekazu Nakamura (nakamura-h) wrote :

I found the fact that from Nova API 2.53, /os-services returns the id of the service as a uuid instead of id of the service.
reference: https://developer.openstack.org/api-ref/compute/#id365

Revision history for this message
Alexander Chadin (joker946) wrote :

nova.hypervisors.list() should return id as uuid for 2.53 (https://developer.openstack.org/api-ref/compute/#list-hypervisors). If we use this uuid to find the specific service (self.nova.find(id=service_id)), then it should work good. Maybe you have old database or you haven't updated nova?

Revision history for this message
Hidekazu Nakamura (nakamura-h) wrote :
Download full text (5.4 KiB)

My multi node devstack environment is re-installed in August 8th.
Bug report is under default watcher which means api_version of nova_client is 2.

After changing api_version of nova_client to 2.53, exception has changed:

8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling Traceback (most recent call last):
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/sc
heduling.py", line 74, in _sync
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling timed_sync()
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/scheduling.py", line 62, in _timed_sync
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling sync_func()
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/base.py", line 185, in synchronize
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling self.cluster_data_model = self.execute()
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 63, in execute
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling return builder.execute()
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 368, in execute
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling self._add_physical_layer()
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 99, in _add_physical_layer
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling self.add_compute_node(cnode)
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 103, in add_compute_node
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling compute_node = self.build_compute_node(node)
 8月 09 22:35:26 w001 watcher-decision-engine[17103]: 2017-08-09 22:35:26.717 TRACE watcher.decision_engine.scheduling File "/opt/stack/watcher/watcher/decision_engine/model/collector/nova.py", line 144, in build_compute_node
 8月 09 22:35:26 w001 watcher-decisio...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to watcher (master)

Reviewed: https://review.openstack.org/499031
Committed: https://git.openstack.org/cgit/openstack/watcher/commit/?id=cf92ece9362232cd35b352ed75929881226b4f9e
Submitter: Jenkins
Branch: master

commit cf92ece9362232cd35b352ed75929881226b4f9e
Author: Hidekazu Nakamura <email address hidden>
Date: Wed Aug 30 14:23:30 2017 +0900

    Update default Nova API version to 2.53(Pike)

    Services are now identified by uuid instead of database id to ensure
    uniqueness across cells.
    GET /os-services returns a uuid in the id field of the response
    from API microversion 2.53(maximum in Pike)

    This patch set updates default Nova API version to 2.53.

    Change-Id: Ib9fefb794eda3c9e75c6a2f5cfdb0e682b8955f3
    Closes-Bug: #1709544

Changed in watcher:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/watcher 1.5.0

This issue was fixed in the openstack/watcher 1.5.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.