| 2021-08-09 11:47:00 |
Adam Tomas |
bug |
|
|
added bug |
| 2021-08-09 11:59:43 |
Adam Tomas |
description |
In kolla-ansible multinode environment I can't add any host to masakari segment.
openstack segment host create compute1 COMPUTE SSH SEGMENT_ID
ends with
BadRequestException: 400: Client Error for url: http://INTERNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
At the same time nova returns proper values for GET/os-services
openstack compute service list
+----+----------------+-------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+-------------+----------+---------+-------+----------------------------+
| 2 | nova-scheduler | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 12 | nova-scheduler | controller2 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 31 | nova-scheduler | controller3 | internal | enabled | up | 2021-08-09T11:08:18.000000 |
| 2 | nova-conductor | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 10 | nova-conductor | controller2 | internal | enabled | up | 2021-08-09T11:08:12.000000 |
| 20 | nova-compute | compute1 | nova | enabled | up | 2021-08-09T11:08:11.000000 |
| 21 | nova-compute | compute2 | nova | enabled | up | 2021-08-09T11:08:16.000000 |
| 25 | nova-conductor | controller3 | internal | enabled | up | 2021-08-09T11:08:17.000000 |
+----+----------------+-------------+----------+---------+-------+----------------------------+
nova service-list
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 1 | nova-scheduler | os-ppu-controller1 | internal | enabled | up | 2021-08-09T11:10:10.000000 | - | False |
| 2 | nova-scheduler | os-ppu-controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 3 | nova-scheduler | os-ppu-controller3 | internal | enabled | up | 2021-08-09T11:10:08.000000 | - | False |
| 4 | nova-conductor | os-ppu-controller1 | internal | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 5 | nova-conductor | os-ppu-controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 6 | nova-compute | os-ppu-compute1 | nova | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
| 7 | nova-compute | os-ppu-compute2 | nova | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 8 | nova-conductor | os-ppu-controller3 | internal | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
masakari-wsgi.log on controller1 (similar on controllers2 and 3)
2021-08-09 08:26:35.639 957 DEBUG masakari.api.openstack.wsgi [req-456c44cd-4bfd-43c7-8697-ede2943418b3 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.625 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8fef7220>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.667141", "type": "VM", "payload": {"event": "IO_ERROR_REASON", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "UNKNOWN"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.641 960 INFO masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.642 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.787 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8ff38940>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.663507", "type": "VM", "payload": {"event": "IO_ERROR", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "IO_ERROR_REPORT"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.804 958 INFO masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.804 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
openstack --debug segment host create os-ppu-compute1 COMPUTE SSH SEGMENT_ID
START with options: --debug segment host create compute1 COMPUTE SSH SEGMENT_ID
(...) OUTPUT OMITTED
Auth plugin password selected
(...) OUTPUT OMITTED
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
ha API version 1.1, cmd group openstack.ha.v1
infra-optim API version 1, cmd group openstack.infra_optim.v1
clustering API version 1, cmd group openstack.clustering.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
rating API version 1, cmd group openstack.rating.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
command: segment host create -> masakariclient.osc.v1.host.CreateHost (auth=True)
Auth plugin password selected
(...) OUTPUT OMITTED
Using auth plugin: password
Using parameters {'auth_url': 'http://INETRNAL_NET_IP:35357/v3', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:35357/v3 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:35357
http://INETRNAL_NET_IP:35357 "GET /v3 HTTP/1.1" 200 255
RESP: [200] content-length: 255 content-type: application/json date: Mon, 09 Aug 2021 11:28:36 GMT server: Apache vary: X-Auth-Token x-openstack-request-id: req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
RESP BODY: {"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:35357/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}
GET call to http://INETRNAL_NET_IP:35357/v3 used request id req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
run(Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[]))
Instantiating masakari service client: <class 'masakariclient.v1.client.Client'>
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:15868 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:15868
http://INETRNAL_NET_IP:15868 "GET / HTTP/1.1" 200 197
RESP: [200] content-length: 197 content-type: application/json date: Mon, 09 Aug 2021 11:28:37 GMT server: Apache
RESP BODY: {"versions": [{"id": "v1.0", "status": "CURRENT", "version": "1.1", "min_version": "1.0", "updated": "2016-07-01T11:33:21Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:15868/v1/"}]}]}
REQ: curl -g -i -X POST http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts -H "Content-Type: application/json" -H "OpenStack-API-Version: instance-ha 1.1" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" -H "X-Auth-Token: {SHA256}c34ac4e56c10325ea90f8e659582a5230ee4901ac7ce1a9a58494c8655eaf202" -d '{"host": {"control_attributes": "SSH", "name": "compute1", "type": "COMPUTE"}}'
http://INETRNAL_NET_IP:15868 "POST /v1/segments/SEGMENT_ID/hosts HTTP/1.1" 400 105
RESP: [400] content-length: 105 content-type: application/json; charset=UTF-8 date: Mon, 09 Aug 2021 11:28:37 GMT openstack-api-version: instance-ha 1.1 server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-298f0f37-6e46-4e5b-95c9-f0aff014d144
RESP BODY: {"badRequest": {"code": 400, "message": "Compute service with name compute1 could not be found."}}
POST call to instance-ha for http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts used request id req-298f0f37-6e46-4e5b-95c9-f0aff014d144
Failed to create segment host: Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[])
BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 117, in run
column_names, data = self.take_action(parsed_args)
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 177, in take_action
raise ex
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 172, in take_action
host = masakari_client.create_host(
File "/usr/lib/python3/dist-packages/openstack/instance_ha/v1/_proxy.py", line 152, in create_host
return self._create(_host.Host, segment_id=segment_id, **attrs)
File "/usr/lib/python3/dist-packages/openstack/proxy.py", line 417, in _create
return res.create(self, base_path=base_path)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1291, in create
self._translate_response(response, has_body=has_body)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1113, in _translate_response
exceptions.raise_from_response(response, error_message=error_message)
File "/usr/lib/python3/dist-packages/openstack/exceptions.py", line 234, in raise_from_response
raise cls(
openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
clean_up CreateHost: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
END return value: 1
Masakari containers built from latest images, the same configuration running in RegionTwo works, but RegionTwo is all-in-one and does not use FQDNs, only IPs. Still - all FQDNs in RegionOne are resolved without any problems (via /etc/hosts on each node).
Masakari was completely removed (deleted containers and images, masakari database and all configuration files) and deployed again with the same result. What else can I do? |
In kolla-ansible multinode environment I can't add any host to masakari segment.
openstack segment host create compute1 COMPUTE SSH SEGMENT_ID
ends with
BadRequestException: 400: Client Error for url: http://INTERNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
At the same time nova returns proper values for GET/os-services
openstack compute service list
+----+----------------+-------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+-------------+----------+---------+-------+----------------------------+
| 2 | nova-scheduler | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 12 | nova-scheduler | controller2 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 31 | nova-scheduler | controller3 | internal | enabled | up | 2021-08-09T11:08:18.000000 |
| 2 | nova-conductor | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 10 | nova-conductor | controller2 | internal | enabled | up | 2021-08-09T11:08:12.000000 |
| 20 | nova-compute | compute1 | nova | enabled | up | 2021-08-09T11:08:11.000000 |
| 21 | nova-compute | compute2 | nova | enabled | up | 2021-08-09T11:08:16.000000 |
| 25 | nova-conductor | controller3 | internal | enabled | up | 2021-08-09T11:08:17.000000 |
+----+----------------+-------------+----------+---------+-------+----------------------------+
nova service-list
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 1 | nova-scheduler | os-ppu-controller1 | internal | enabled | up | 2021-08-09T11:10:10.000000 | - | False |
| 2 | nova-scheduler | os-ppu-controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 3 | nova-scheduler | os-ppu-controller3 | internal | enabled | up | 2021-08-09T11:10:08.000000 | - | False |
| 4 | nova-conductor | os-ppu-controller1 | internal | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 5 | nova-conductor | os-ppu-controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 6 | nova-compute | os-ppu-compute1 | nova | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
| 7 | nova-compute | os-ppu-compute2 | nova | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 8 | nova-conductor | os-ppu-controller3 | internal | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
masakari-wsgi.log on controller1 (similar on controllers2 and 3)
2021-08-09 08:26:35.639 957 DEBUG masakari.api.openstack.wsgi [req-456c44cd-4bfd-43c7-8697-ede2943418b3 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.625 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8fef7220>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.667141", "type": "VM", "payload": {"event": "IO_ERROR_REASON", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "UNKNOWN"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.641 960 INFO masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.642 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.787 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8ff38940>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.663507", "type": "VM", "payload": {"event": "IO_ERROR", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "IO_ERROR_REPORT"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.804 958 INFO masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.804 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
openstack --debug segment host create compute1 COMPUTE SSH SEGMENT_ID
START with options: --debug segment host create compute1 COMPUTE SSH SEGMENT_ID
(...) OUTPUT OMITTED
Auth plugin password selected
(...) OUTPUT OMITTED
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
ha API version 1.1, cmd group openstack.ha.v1
infra-optim API version 1, cmd group openstack.infra_optim.v1
clustering API version 1, cmd group openstack.clustering.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
rating API version 1, cmd group openstack.rating.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
command: segment host create -> masakariclient.osc.v1.host.CreateHost (auth=True)
Auth plugin password selected
(...) OUTPUT OMITTED
Using auth plugin: password
Using parameters {'auth_url': 'http://INETRNAL_NET_IP:35357/v3', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:35357/v3 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:35357
http://INETRNAL_NET_IP:35357 "GET /v3 HTTP/1.1" 200 255
RESP: [200] content-length: 255 content-type: application/json date: Mon, 09 Aug 2021 11:28:36 GMT server: Apache vary: X-Auth-Token x-openstack-request-id: req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
RESP BODY: {"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:35357/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}
GET call to http://INETRNAL_NET_IP:35357/v3 used request id req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
run(Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[]))
Instantiating masakari service client: <class 'masakariclient.v1.client.Client'>
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:15868 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:15868
http://INETRNAL_NET_IP:15868 "GET / HTTP/1.1" 200 197
RESP: [200] content-length: 197 content-type: application/json date: Mon, 09 Aug 2021 11:28:37 GMT server: Apache
RESP BODY: {"versions": [{"id": "v1.0", "status": "CURRENT", "version": "1.1", "min_version": "1.0", "updated": "2016-07-01T11:33:21Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:15868/v1/"}]}]}
REQ: curl -g -i -X POST http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts -H "Content-Type: application/json" -H "OpenStack-API-Version: instance-ha 1.1" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" -H "X-Auth-Token: {SHA256}c34ac4e56c10325ea90f8e659582a5230ee4901ac7ce1a9a58494c8655eaf202" -d '{"host": {"control_attributes": "SSH", "name": "compute1", "type": "COMPUTE"}}'
http://INETRNAL_NET_IP:15868 "POST /v1/segments/SEGMENT_ID/hosts HTTP/1.1" 400 105
RESP: [400] content-length: 105 content-type: application/json; charset=UTF-8 date: Mon, 09 Aug 2021 11:28:37 GMT openstack-api-version: instance-ha 1.1 server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-298f0f37-6e46-4e5b-95c9-f0aff014d144
RESP BODY: {"badRequest": {"code": 400, "message": "Compute service with name compute1 could not be found."}}
POST call to instance-ha for http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts used request id req-298f0f37-6e46-4e5b-95c9-f0aff014d144
Failed to create segment host: Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[])
BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 117, in run
column_names, data = self.take_action(parsed_args)
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 177, in take_action
raise ex
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 172, in take_action
host = masakari_client.create_host(
File "/usr/lib/python3/dist-packages/openstack/instance_ha/v1/_proxy.py", line 152, in create_host
return self._create(_host.Host, segment_id=segment_id, **attrs)
File "/usr/lib/python3/dist-packages/openstack/proxy.py", line 417, in _create
return res.create(self, base_path=base_path)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1291, in create
self._translate_response(response, has_body=has_body)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1113, in _translate_response
exceptions.raise_from_response(response, error_message=error_message)
File "/usr/lib/python3/dist-packages/openstack/exceptions.py", line 234, in raise_from_response
raise cls(
openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
clean_up CreateHost: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
END return value: 1
Masakari containers built from latest images, the same configuration running in RegionTwo works, but RegionTwo is all-in-one and does not use FQDNs, only IPs. Still - all FQDNs in RegionOne are resolved without any problems (via /etc/hosts on each node).
Masakari was completely removed (deleted containers and images, masakari database and all configuration files) and deployed again with the same result. What else can I do? |
|
| 2021-08-09 21:49:54 |
Adam Tomas |
description |
In kolla-ansible multinode environment I can't add any host to masakari segment.
openstack segment host create compute1 COMPUTE SSH SEGMENT_ID
ends with
BadRequestException: 400: Client Error for url: http://INTERNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
At the same time nova returns proper values for GET/os-services
openstack compute service list
+----+----------------+-------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+-------------+----------+---------+-------+----------------------------+
| 2 | nova-scheduler | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 12 | nova-scheduler | controller2 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 31 | nova-scheduler | controller3 | internal | enabled | up | 2021-08-09T11:08:18.000000 |
| 2 | nova-conductor | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 10 | nova-conductor | controller2 | internal | enabled | up | 2021-08-09T11:08:12.000000 |
| 20 | nova-compute | compute1 | nova | enabled | up | 2021-08-09T11:08:11.000000 |
| 21 | nova-compute | compute2 | nova | enabled | up | 2021-08-09T11:08:16.000000 |
| 25 | nova-conductor | controller3 | internal | enabled | up | 2021-08-09T11:08:17.000000 |
+----+----------------+-------------+----------+---------+-------+----------------------------+
nova service-list
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 1 | nova-scheduler | os-ppu-controller1 | internal | enabled | up | 2021-08-09T11:10:10.000000 | - | False |
| 2 | nova-scheduler | os-ppu-controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 3 | nova-scheduler | os-ppu-controller3 | internal | enabled | up | 2021-08-09T11:10:08.000000 | - | False |
| 4 | nova-conductor | os-ppu-controller1 | internal | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 5 | nova-conductor | os-ppu-controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 6 | nova-compute | os-ppu-compute1 | nova | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
| 7 | nova-compute | os-ppu-compute2 | nova | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 8 | nova-conductor | os-ppu-controller3 | internal | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
masakari-wsgi.log on controller1 (similar on controllers2 and 3)
2021-08-09 08:26:35.639 957 DEBUG masakari.api.openstack.wsgi [req-456c44cd-4bfd-43c7-8697-ede2943418b3 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.625 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8fef7220>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.667141", "type": "VM", "payload": {"event": "IO_ERROR_REASON", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "UNKNOWN"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.641 960 INFO masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.642 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.787 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8ff38940>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.663507", "type": "VM", "payload": {"event": "IO_ERROR", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "IO_ERROR_REPORT"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.804 958 INFO masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.804 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
openstack --debug segment host create compute1 COMPUTE SSH SEGMENT_ID
START with options: --debug segment host create compute1 COMPUTE SSH SEGMENT_ID
(...) OUTPUT OMITTED
Auth plugin password selected
(...) OUTPUT OMITTED
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
ha API version 1.1, cmd group openstack.ha.v1
infra-optim API version 1, cmd group openstack.infra_optim.v1
clustering API version 1, cmd group openstack.clustering.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
rating API version 1, cmd group openstack.rating.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
command: segment host create -> masakariclient.osc.v1.host.CreateHost (auth=True)
Auth plugin password selected
(...) OUTPUT OMITTED
Using auth plugin: password
Using parameters {'auth_url': 'http://INETRNAL_NET_IP:35357/v3', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:35357/v3 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:35357
http://INETRNAL_NET_IP:35357 "GET /v3 HTTP/1.1" 200 255
RESP: [200] content-length: 255 content-type: application/json date: Mon, 09 Aug 2021 11:28:36 GMT server: Apache vary: X-Auth-Token x-openstack-request-id: req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
RESP BODY: {"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:35357/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}
GET call to http://INETRNAL_NET_IP:35357/v3 used request id req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
run(Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[]))
Instantiating masakari service client: <class 'masakariclient.v1.client.Client'>
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:15868 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:15868
http://INETRNAL_NET_IP:15868 "GET / HTTP/1.1" 200 197
RESP: [200] content-length: 197 content-type: application/json date: Mon, 09 Aug 2021 11:28:37 GMT server: Apache
RESP BODY: {"versions": [{"id": "v1.0", "status": "CURRENT", "version": "1.1", "min_version": "1.0", "updated": "2016-07-01T11:33:21Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:15868/v1/"}]}]}
REQ: curl -g -i -X POST http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts -H "Content-Type: application/json" -H "OpenStack-API-Version: instance-ha 1.1" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" -H "X-Auth-Token: {SHA256}c34ac4e56c10325ea90f8e659582a5230ee4901ac7ce1a9a58494c8655eaf202" -d '{"host": {"control_attributes": "SSH", "name": "compute1", "type": "COMPUTE"}}'
http://INETRNAL_NET_IP:15868 "POST /v1/segments/SEGMENT_ID/hosts HTTP/1.1" 400 105
RESP: [400] content-length: 105 content-type: application/json; charset=UTF-8 date: Mon, 09 Aug 2021 11:28:37 GMT openstack-api-version: instance-ha 1.1 server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-298f0f37-6e46-4e5b-95c9-f0aff014d144
RESP BODY: {"badRequest": {"code": 400, "message": "Compute service with name compute1 could not be found."}}
POST call to instance-ha for http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts used request id req-298f0f37-6e46-4e5b-95c9-f0aff014d144
Failed to create segment host: Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[])
BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 117, in run
column_names, data = self.take_action(parsed_args)
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 177, in take_action
raise ex
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 172, in take_action
host = masakari_client.create_host(
File "/usr/lib/python3/dist-packages/openstack/instance_ha/v1/_proxy.py", line 152, in create_host
return self._create(_host.Host, segment_id=segment_id, **attrs)
File "/usr/lib/python3/dist-packages/openstack/proxy.py", line 417, in _create
return res.create(self, base_path=base_path)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1291, in create
self._translate_response(response, has_body=has_body)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1113, in _translate_response
exceptions.raise_from_response(response, error_message=error_message)
File "/usr/lib/python3/dist-packages/openstack/exceptions.py", line 234, in raise_from_response
raise cls(
openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
clean_up CreateHost: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
END return value: 1
Masakari containers built from latest images, the same configuration running in RegionTwo works, but RegionTwo is all-in-one and does not use FQDNs, only IPs. Still - all FQDNs in RegionOne are resolved without any problems (via /etc/hosts on each node).
Masakari was completely removed (deleted containers and images, masakari database and all configuration files) and deployed again with the same result. What else can I do? |
In kolla-ansible multinode environment I can't add any host to masakari segment.
openstack segment host create compute1 COMPUTE SSH SEGMENT_ID
ends with
BadRequestException: 400: Client Error for url: http://INTERNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
At the same time nova returns proper values for GET/os-services
openstack compute service list
+----+----------------+-------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+-------------+----------+---------+-------+----------------------------+
| 2 | nova-scheduler | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 12 | nova-scheduler | controller2 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 31 | nova-scheduler | controller3 | internal | enabled | up | 2021-08-09T11:08:18.000000 |
| 2 | nova-conductor | controller1 | internal | enabled | up | 2021-08-09T11:08:16.000000 |
| 10 | nova-conductor | controller2 | internal | enabled | up | 2021-08-09T11:08:12.000000 |
| 20 | nova-compute | compute1 | nova | enabled | up | 2021-08-09T11:08:11.000000 |
| 21 | nova-compute | compute2 | nova | enabled | up | 2021-08-09T11:08:16.000000 |
| 25 | nova-conductor | controller3 | internal | enabled | up | 2021-08-09T11:08:17.000000 |
+----+----------------+-------------+----------+---------+-------+----------------------------+
nova service-list
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 1 | nova-scheduler | controller1 | internal | enabled | up | 2021-08-09T11:10:10.000000 | - | False |
| 2 | nova-scheduler | controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 3 | nova-scheduler | controller3 | internal | enabled | up | 2021-08-09T11:10:08.000000 | - | False |
| 4 | nova-conductor | controller1 | internal | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 5 | nova-conductor | controller2 | internal | enabled | up | 2021-08-09T11:10:12.000000 | - | False |
| 6 | nova-compute | compute1 | nova | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
| 7 | nova-compute | compute2 | nova | enabled | up | 2021-08-09T11:10:06.000000 | - | False |
| 8 | nova-conductor | controller3 | internal | enabled | up | 2021-08-09T11:10:11.000000 | - | False |
+--------------------------------------+----------------+--------------------+----------+---------+-------+----------------------------+-----------------+-------------+
masakari-wsgi.log on controller1 (similar on controllers2 and 3)
2021-08-09 08:26:35.639 957 DEBUG masakari.api.openstack.wsgi [req-456c44cd-4bfd-43c7-8697-ede2943418b3 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.625 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8fef7220>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.667141", "type": "VM", "payload": {"event": "IO_ERROR_REASON", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "UNKNOWN"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.641 960 INFO masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.642 960 DEBUG masakari.api.openstack.wsgi [req-21c1c424-5725-4661-a0aa-c70f4dbb6895 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
2021-08-09 08:26:45.787 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Action: 'create', calling method: <bound method NotificationsController.create of <masakari.api.openstack.ha.notifications.NotificationsController object at 0x7fbe8ff38940>>, body: {"notification": {"hostname": "compute1", "generated_time": "2021-08-09T08:25:14.663507", "type": "VM", "payload": {"event": "IO_ERROR", "instance_uuid": "025e08fa-9606-4bcd-a398-b6ebea9b3adb", "vir_domain_event": "IO_ERROR_REPORT"}}} _process_stack /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:619
2021-08-09 08:26:45.804 958 INFO masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] HTTP exception thrown: Host with name compute1 could not be found.
2021-08-09 08:26:45.804 958 DEBUG masakari.api.openstack.wsgi [req-a48afd1f-60be-4a7f-892b-ecbea4309fe9 9fea0ade15644c78b5a1a4166b0256f7 775839ee714f4dfa85975b4b61adc778 - - -] Returning 400 to user: Host with name compute1 could not be found. __call__ /var/lib/kolla/venv/lib/python3.8/site-packages/masakari/api/openstack/wsgi.py:1038
openstack --debug segment host create compute1 COMPUTE SSH SEGMENT_ID
START with options: --debug segment host create compute1 COMPUTE SSH SEGMENT_ID
(...) OUTPUT OMITTED
Auth plugin password selected
(...) OUTPUT OMITTED
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
ha API version 1.1, cmd group openstack.ha.v1
infra-optim API version 1, cmd group openstack.infra_optim.v1
clustering API version 1, cmd group openstack.clustering.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
rating API version 1, cmd group openstack.rating.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
command: segment host create -> masakariclient.osc.v1.host.CreateHost (auth=True)
Auth plugin password selected
(...) OUTPUT OMITTED
Using auth plugin: password
Using parameters {'auth_url': 'http://INETRNAL_NET_IP:35357/v3', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:35357/v3 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:35357
http://INETRNAL_NET_IP:35357 "GET /v3 HTTP/1.1" 200 255
RESP: [200] content-length: 255 content-type: application/json date: Mon, 09 Aug 2021 11:28:36 GMT server: Apache vary: X-Auth-Token x-openstack-request-id: req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
RESP BODY: {"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:35357/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}
GET call to http://INETRNAL_NET_IP:35357/v3 used request id req-7036bb8e-4033-4194-8cef-e1b00f5c7b08
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
run(Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[]))
Instantiating masakari service client: <class 'masakariclient.v1.client.Client'>
Making authentication request to http://INETRNAL_NET_IP:35357/v3/auth/tokens
http://INETRNAL_NET_IP:35357 "POST /v3/auth/tokens HTTP/1.1" 201 25266
(...) OUTPUT OMITTED
REQ: curl -g -i -X GET http://INETRNAL_NET_IP:15868 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"
Starting new HTTP connection (1): INETRNAL_NET_IP:15868
http://INETRNAL_NET_IP:15868 "GET / HTTP/1.1" 200 197
RESP: [200] content-length: 197 content-type: application/json date: Mon, 09 Aug 2021 11:28:37 GMT server: Apache
RESP BODY: {"versions": [{"id": "v1.0", "status": "CURRENT", "version": "1.1", "min_version": "1.0", "updated": "2016-07-01T11:33:21Z", "links": [{"rel": "self", "href": "http://INETRNAL_NET_IP:15868/v1/"}]}]}
REQ: curl -g -i -X POST http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts -H "Content-Type: application/json" -H "OpenStack-API-Version: instance-ha 1.1" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" -H "X-Auth-Token: {SHA256}c34ac4e56c10325ea90f8e659582a5230ee4901ac7ce1a9a58494c8655eaf202" -d '{"host": {"control_attributes": "SSH", "name": "compute1", "type": "COMPUTE"}}'
http://INETRNAL_NET_IP:15868 "POST /v1/segments/SEGMENT_ID/hosts HTTP/1.1" 400 105
RESP: [400] content-length: 105 content-type: application/json; charset=UTF-8 date: Mon, 09 Aug 2021 11:28:37 GMT openstack-api-version: instance-ha 1.1 server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-298f0f37-6e46-4e5b-95c9-f0aff014d144
RESP BODY: {"badRequest": {"code": 400, "message": "Compute service with name compute1 could not be found."}}
POST call to instance-ha for http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts used request id req-298f0f37-6e46-4e5b-95c9-f0aff014d144
Failed to create segment host: Namespace(columns=[], control_attributes='SSH', fit_width=False, formatter='table', max_width=0, name='compute1', noindent=False, on_maintenance=None, prefix='', print_empty=False, reserved=None, segment_id='SEGMENT_ID', type='COMPUTE', variables=[])
BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 117, in run
column_names, data = self.take_action(parsed_args)
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 177, in take_action
raise ex
File "/usr/local/lib/python3.8/dist-packages/masakariclient/osc/v1/host.py", line 172, in take_action
host = masakari_client.create_host(
File "/usr/lib/python3/dist-packages/openstack/instance_ha/v1/_proxy.py", line 152, in create_host
return self._create(_host.Host, segment_id=segment_id, **attrs)
File "/usr/lib/python3/dist-packages/openstack/proxy.py", line 417, in _create
return res.create(self, base_path=base_path)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1291, in create
self._translate_response(response, has_body=has_body)
File "/usr/lib/python3/dist-packages/openstack/resource.py", line 1113, in _translate_response
exceptions.raise_from_response(response, error_message=error_message)
File "/usr/lib/python3/dist-packages/openstack/exceptions.py", line 234, in raise_from_response
raise cls(
openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
clean_up CreateHost: BadRequestException: 400: Client Error for url: http://INETRNAL_NET_IP:15868/v1/segments/SEGMENT_ID/hosts, Compute service with name compute1 could not be found.
END return value: 1
Masakari containers built from latest images, the same configuration running in RegionTwo works, but RegionTwo is all-in-one and does not use FQDNs, only IPs. Still - all FQDNs in RegionOne are resolved without any problems (via /etc/hosts on each node).
Masakari was completely removed (deleted containers and images, masakari database and all configuration files) and deployed again with the same result. What else can I do? |
|
| 2021-08-18 18:49:41 |
Radosław Piliszek |
summary |
Can't add host to masakari segment - "Compute service with name XXXX could not be found" |
Masakari-Nova region name unset |
|
| 2021-08-18 18:49:47 |
Radosław Piliszek |
masakari: status |
New |
Invalid |
|
| 2021-08-18 18:49:57 |
Radosław Piliszek |
bug task added |
|
kolla-ansible |
|
| 2021-08-18 18:50:02 |
Radosław Piliszek |
kolla-ansible: status |
New |
Triaged |
|
| 2021-08-18 18:50:05 |
Radosław Piliszek |
kolla-ansible: importance |
Undecided |
Medium |
|
| 2021-08-18 18:52:07 |
Radosław Piliszek |
kolla-ansible: assignee |
|
Radosław Piliszek (yoctozepto) |
|
| 2021-08-20 18:54:51 |
OpenStack Infra |
kolla-ansible: status |
Triaged |
In Progress |
|
| 2021-08-27 08:55:10 |
OpenStack Infra |
kolla-ansible: status |
In Progress |
Fix Released |
|
| 2021-08-31 14:06:07 |
OpenStack Infra |
tags |
masakari |
in-stable-victoria masakari |
|
| 2021-08-31 14:06:13 |
OpenStack Infra |
tags |
in-stable-victoria masakari |
in-stable-victoria in-stable-wallaby masakari |
|
