Unclear 'dns service' statuses and states; need to upgrade configuration
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Designate |
Triaged
|
High
|
Unassigned |
Bug Description
Hello!
Recently I've deployed Designate on two nodes and started to check/test it via cli (python-
root@control01:~# openstack dns service list
+------
| id | hostname | service_name | status | stats | capabilities |
+------
| 3871fdbf-
| aafee96e-
| 001140fc-
| bb105334-
| dc0fe0fb-
| 7237e35d-
| 8aeb091e-
| f126bb73-
+------
[1] Output shows not all of services actually installed on the nodes. In my case, I've also have designate-producer service, which is missed in the table.
[2] 'Status' column is not exactly service status - 'active', 'stopped', 'failed', etc. It looks like 'enabled/disabled' status, because even I've stopped all designate services on two nodes, command always shows me status 'UP'...
For example, this otput is looking much clearly:
root@control01:~# openstack compute service list
+----+-
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+-
| 2 | nova-scheduler | control02-api | internal | enabled | up | 2018-04-
| 4 | nova-scheduler | control01-api | internal | enabled | up | 2018-04-
| 5 | nova-consoleauth | control01-api | internal | enabled | up | 2018-04-
| 6 | nova-consoleauth | control02-api | internal | enabled | down | 2018-04-
| 17 | nova-conductor | control01-api | internal | enabled | up | 2018-04-
| 19 | nova-conductor | control02-api | internal | enabled | up | 2018-04-
| 20 | nova-compute | compute02-api | nova | enabled | up | 2018-04-
| 23 | nova-compute | compute01-api | nova | enabled | up | 2018-04-
+----+-
From [2] comes that:
[a] we need to update configuration with 'service_down_time' option (Maximum time in seconds since last check-in for up service) and related 'report_interval' option (Number of seconds indicating how frequently the state of services is reported) in [DEFAULT] section
[b] we need to extend 'service_statuses' database table with column 'state', which value is depending on 'service_down_time' value
summary: |
- Unclear 'dns service' statuses and states; need to upgraded - configuration + Unclear 'dns service' statuses and states; need to upgrade configuration |
Changed in designate: | |
status: | New → Triaged |
importance: | Undecided → High |
tags: | added: canonical-bootstack |
The resolution of this bug is blocking the implementation of a smart check for the Designate services in the openstack- service- checks charm
https:/ /bugs.launchpad .net/charm- openstack- service- checks/ +bug/1845293
We can still reproduce the bug in cloud:bionic-rocky:
All services are up:
$ date
Tue Sep 24 11:01:10 UTC 2019
$ openstack dns service list ------- ------- ------- ------- ----+-- ------- ------+ ------- ------- +------ --+---- ---+--- ------- ----+ ------- ------- ------- ------- ----+-- ------- ------+ ------- ------- +------ --+---- ---+--- ------- ----+ 6be5-4f0a- bed6-14ace02f91 ef | juju-e719f5-9 | api | UP | - | - | 23a1-430f- a5b2-0bd5f9a8f3 e8 | juju-e719f5-9 | producer | UP | - | - | 6ecb-4a31- 82a9-814948c5f6 67 | juju-e719f5-9 | mdns | UP | - | - | ------- ------- ------- ------- ----+-- ------- ------+ ------- ------- +------ --+---- ---+--- ------- ----+
+------
| id | hostname | service_name | status | stats | capabilities |
+------
| bac409d2-
| f1b39c47-
| 6103d503-
+------
We stopped the mdns service and after 30 minutes it is still reported as UP even if the heartbeat is 30 minutes old:
$ date
Tue Sep 24 11:36:12 UTC 2019
$ openstack dns service show 6103d503- 6ecb-4a31- 82a9-814948c5f6 67 ------- ---+--- ------- ------- ------- ------- ------- + ------- ---+--- ------- ------- ------- ------- ------- + 23T16:24: 48.000000 | 24T11:06: 03.000000 | 6ecb-4a31- 82a9-814948c5f6 67 | 24T11:06: 03.000000 | ------- ---+--- ------- ------- ------- ------- ------- +
+------
| Field | Value |
+------
| capabilities | - |
| created_at | 2019-09-
| heartbeated_at | 2019-09-
| hostname | juju-e719f5-9 |
| id | 6103d503-
| service_name | mdns |
| stats | - |
| status | UP |
| updated_at | 2019-09-
+------
Is there any news on the resolution of it?