Activity log for bug #1768824

Date Who What changed Old value New value Message
2018-05-03 12:07:27 Dmitry Galkin bug added bug
2018-05-14 20:14:20 Erik Olof Gunnar Andersson designate: status New Confirmed
2018-05-14 20:15:24 Erik Olof Gunnar Andersson designate: importance Undecided High
2018-05-15 05:58:18 OpenStack Infra designate: status Confirmed In Progress
2018-05-15 05:58:18 OpenStack Infra designate: assignee Erik Olof Gunnar Andersson (eandersson)
2018-06-14 11:26:52 OpenStack Infra designate: status In Progress Fix Released
2019-06-03 09:21:41 Edward Hope-Morley description Hi, The service_statuses table in Designate database is running full of records in our deployment: MariaDB [designate]> select count(*) from service_statuses; +----------+ | count(*) | +----------+ | 24474342 | +----------+ 1 row in set (7 min 19.09 sec) We got millions of rows in just couple of month. The problem is that the same services running on the same hosts create new record (instead of updating existing) during status report to Designate. This is how it looks in DB: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | 0dde2b5f228549d5995cb0338841bd50 | 2018-05-02 12:06:03 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:06:03 | UP | {} | {} | | 0e311d3000d8403d97066eba619490a3 | 2018-05-02 12:05:14 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:13 | UP | {} | {} | | 168448cd97cd428ea19318243570482c | 2018-05-02 12:05:48 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:48 | UP | {} | {} | | 1685d7f80d8c4f75b052680e5e2f40ae | 2018-05-02 12:05:59 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:58 | UP | {} | {} | | 192275eb33854b4091b981b0c32d04f7 | 2018-05-02 12:05:41 | NULL | worker | designate-worker-3446544-7fzqx | 2018-05-02 12:05:35 | UP | {} | {} | | 1e465011f21f47f096b54005675e8011 | 2018-05-02 12:05:25 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:05:25 | UP | {} | {} | | 22e0ab87b3cd4228bc191e49923d13ba | 2018-05-02 12:05:58 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:58 | UP | {} | {} | | 284d6b6bf53f47358fa14750d0c2a181 | 2018-05-02 12:05:18 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:18 | UP | {} | {} | | 3617ac740b9945c09f65f4b8cf0a72b5 | 2018-05-02 12:05:25 | NULL | api | designate-api-2042646259-drzws | 2018-05-02 12:05:25 | UP | {} | {} | | 3880157829f946398159f104b13a066b | 2018-05-02 12:06:00 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:06:00 | UP | {} | {} | The respective part of the code is: https://github.com/openstack/designate/blob/master/designate/central/service.py#L2879-L2898 It is not exactly clear to me why the id is needed to identify the service. The 'hostname' and 'service_name' seem to be enough - it is rather unlikely that there are two api services are running on the same host.. So, I've removed the id part: if service_status.obj_attr_is_set('id'): criterion["id"] = service_status.id and cleaned up the service_statuses table, now it looks fine: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | 1e89d10b0ddb459f8a9522ebd62fa629 | 2018-05-02 13:03:21 | 2018-05-02 13:03:37 | worker | designate-worker-2851226431-9274c | 2018-05-02 13:03:36 | UP | {} | {} | | 8d29d50270ee48999b78e90c1b88acf8 | 2018-05-02 13:03:23 | 2018-05-02 13:03:38 | api | designate-api-1699569558-9rvg6 | 2018-05-02 13:03:38 | UP | {} | {} | | 9602ccf55d204c7b86277e21af3d739f | 2018-05-02 13:03:18 | 2018-05-02 13:03:38 | producer | designate-producer-3892623035-6vwt0 | 2018-05-02 13:03:38 | UP | {} | {} | | cec705e082c542cdbfa83b9ed0c2b438 | 2018-05-02 13:03:19 | 2018-05-02 13:03:39 | api | designate-api-1699569558-r018m | 2018-05-02 13:03:39 | UP | {} | {} | | ddd48f13b67448d2949ced39d76e2958 | 2018-05-02 13:03:20 | 2018-05-02 13:03:40 | central | designate-central-2632925199-wwj7j | 2018-05-02 13:03:40 | UP | {} | {} | | ef0ae921a78d4ac090dc74a1dc686921 | 2018-05-02 13:03:21 | 2018-05-02 13:03:41 | mdns | designate-mdns-4162010419-pvclv | 2018-05-02 13:03:41 | UP | {} | {} | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ 6 rows in set (0.00 sec) Our Designate version is Pike, but the code seems to be the same in master branch. [Impact] This patch is required to prevent pool-manager from creating unbounded amounts of status logs in the service_statuses table triggered by having > 1 log in there. [Test Case] * deploy openstack queens with designate * mysql> select count(*) from service_statuses where service_name="pool_manager"; should return 1 * try to add an extra entry: ts=date '+%Y-%m-%d %H:%M:%S' svc_host=`mysql -B -h$host -u${service} -p$passwd ${service} -e 'select hostname from service_statuses where service_name="pool_manager";'| tail -n 1` mysql -h$host -u${service} -p$passwd ${service} -e "insert into service_statuses values ('1234', '$ts', '$ts', 'pool_manager', '$svc_host', '$ts', 'UP', '{}', '{}');" * this should fail since the hostname/servicename columns should now be a unique contraint * can also check this with: mysql -h$host -u${service} -p$passwd ${service} -e "select * from INFORMATION_SCHEMA.TABLE_CONSTRAINTS where CONSTRAINT_TYPE='UNIQUE';"| grep service_statuses [Regression Potential] if the table already has multiple records for pool_manager in the service_statuses table, it will be necessary to (manually) delete all but one record in order for the upgrade to succeed. ---------------------------------------------------------------------------- Hi, The service_statuses table in Designate database is running full of records in our deployment: MariaDB [designate]> select count(*) from service_statuses; +----------+ | count(*) | +----------+ | 24474342 | +----------+ 1 row in set (7 min 19.09 sec) We got millions of rows in just couple of month. The problem is that the same services running on the same hosts create new record (instead of updating existing) during status report to Designate. This is how it looks in DB: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | 0dde2b5f228549d5995cb0338841bd50 | 2018-05-02 12:06:03 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:06:03 | UP | {} | {} | | 0e311d3000d8403d97066eba619490a3 | 2018-05-02 12:05:14 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:13 | UP | {} | {} | | 168448cd97cd428ea19318243570482c | 2018-05-02 12:05:48 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:48 | UP | {} | {} | | 1685d7f80d8c4f75b052680e5e2f40ae | 2018-05-02 12:05:59 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:58 | UP | {} | {} | | 192275eb33854b4091b981b0c32d04f7 | 2018-05-02 12:05:41 | NULL | worker | designate-worker-3446544-7fzqx | 2018-05-02 12:05:35 | UP | {} | {} | | 1e465011f21f47f096b54005675e8011 | 2018-05-02 12:05:25 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:05:25 | UP | {} | {} | | 22e0ab87b3cd4228bc191e49923d13ba | 2018-05-02 12:05:58 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:58 | UP | {} | {} | | 284d6b6bf53f47358fa14750d0c2a181 | 2018-05-02 12:05:18 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:18 | UP | {} | {} | | 3617ac740b9945c09f65f4b8cf0a72b5 | 2018-05-02 12:05:25 | NULL | api | designate-api-2042646259-drzws | 2018-05-02 12:05:25 | UP | {} | {} | | 3880157829f946398159f104b13a066b | 2018-05-02 12:06:00 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:06:00 | UP | {} | {} | The respective part of the code is: https://github.com/openstack/designate/blob/master/designate/central/service.py#L2879-L2898 It is not exactly clear to me why the id is needed to identify the service. The 'hostname' and 'service_name' seem to be enough - it is rather unlikely that there are two api services are running on the same host.. So, I've removed the id part:         if service_status.obj_attr_is_set('id'):             criterion["id"] = service_status.id and cleaned up the service_statuses table, now it looks fine: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | 1e89d10b0ddb459f8a9522ebd62fa629 | 2018-05-02 13:03:21 | 2018-05-02 13:03:37 | worker | designate-worker-2851226431-9274c | 2018-05-02 13:03:36 | UP | {} | {} | | 8d29d50270ee48999b78e90c1b88acf8 | 2018-05-02 13:03:23 | 2018-05-02 13:03:38 | api | designate-api-1699569558-9rvg6 | 2018-05-02 13:03:38 | UP | {} | {} | | 9602ccf55d204c7b86277e21af3d739f | 2018-05-02 13:03:18 | 2018-05-02 13:03:38 | producer | designate-producer-3892623035-6vwt0 | 2018-05-02 13:03:38 | UP | {} | {} | | cec705e082c542cdbfa83b9ed0c2b438 | 2018-05-02 13:03:19 | 2018-05-02 13:03:39 | api | designate-api-1699569558-r018m | 2018-05-02 13:03:39 | UP | {} | {} | | ddd48f13b67448d2949ced39d76e2958 | 2018-05-02 13:03:20 | 2018-05-02 13:03:40 | central | designate-central-2632925199-wwj7j | 2018-05-02 13:03:40 | UP | {} | {} | | ef0ae921a78d4ac090dc74a1dc686921 | 2018-05-02 13:03:21 | 2018-05-02 13:03:41 | mdns | designate-mdns-4162010419-pvclv | 2018-05-02 13:03:41 | UP | {} | {} | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ 6 rows in set (0.00 sec) Our Designate version is Pike, but the code seems to be the same in master branch.
2019-06-03 09:21:49 Edward Hope-Morley summary service_statuses table running full in Designate database [SRU] service_statuses table running full in Designate database
2019-06-03 09:22:01 Edward Hope-Morley bug task added cloud-archive
2019-06-03 09:22:15 Edward Hope-Morley nominated for series cloud-archive/queens
2019-06-03 09:22:15 Edward Hope-Morley bug task added cloud-archive/queens
2019-06-03 09:22:21 Edward Hope-Morley cloud-archive: status New Fix Released
2019-06-03 14:22:09 Edward Hope-Morley attachment added lp1606741-rocky.debdiff https://bugs.launchpad.net/cloud-archive/+bug/1768824/+attachment/5268542/+files/lp1606741-rocky.debdiff
2019-06-03 14:23:35 Edward Hope-Morley bug task added designate (Ubuntu)
2019-06-03 14:23:42 Edward Hope-Morley designate (Ubuntu): status New Fix Released
2019-06-03 14:23:47 Edward Hope-Morley nominated for series Ubuntu Bionic
2019-06-03 14:23:47 Edward Hope-Morley bug task added designate (Ubuntu Bionic)
2019-06-03 14:31:03 Edward Hope-Morley attachment removed lp1606741-queens.debdiff https://bugs.launchpad.net/designate/+bug/1768824/+attachment/5268542/+files/lp1606741-rocky.debdiff
2019-06-03 14:32:04 Edward Hope-Morley attachment added lp1768824-bionic-queens.debdiff https://bugs.launchpad.net/designate/+bug/1768824/+attachment/5268554/+files/lp1768824-bionic-queens.debdiff
2019-06-04 16:55:27 Edward Hope-Morley tags sts sts-sru-needed
2019-06-24 13:55:21 Corey Bryant cloud-archive/queens: status New Triaged
2019-06-24 13:55:24 Corey Bryant cloud-archive/queens: importance Undecided High
2019-06-24 13:55:27 Corey Bryant designate (Ubuntu Bionic): status New Triaged
2019-06-24 13:55:29 Corey Bryant designate (Ubuntu Bionic): importance Undecided High
2019-07-08 18:55:53 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2019-07-10 20:07:22 Brian Murray designate (Ubuntu Bionic): status Triaged Fix Committed
2019-07-10 20:07:25 Brian Murray bug added subscriber SRU Verification
2019-07-10 20:07:31 Brian Murray tags sts sts-sru-needed sts sts-sru-needed verification-needed verification-needed-bionic
2019-07-12 10:29:50 Edward Hope-Morley attachment removed lp1768824-bionic-queens.debdiff https://bugs.launchpad.net/cloud-archive/+bug/1768824/+attachment/5268554/+files/lp1768824-bionic-queens.debdiff
2019-07-12 10:30:06 Edward Hope-Morley attachment added lp1768824-bionic.debdiff https://bugs.launchpad.net/cloud-archive/+bug/1768824/+attachment/5276673/+files/lp1768824-bionic.debdiff
2019-07-12 10:30:35 Edward Hope-Morley tags sts sts-sru-needed verification-needed verification-needed-bionic sts sts-sru-needed verification-failed verification-failed-bionic
2019-07-15 15:21:57 Łukasz Zemczak tags sts sts-sru-needed verification-failed verification-failed-bionic sts sts-sru-needed verification-needed verification-needed-bionic
2019-07-22 13:52:37 Corey Bryant cloud-archive/queens: status Triaged Fix Committed
2019-07-22 13:52:39 Corey Bryant tags sts sts-sru-needed verification-needed verification-needed-bionic sts sts-sru-needed verification-needed verification-needed-bionic verification-queens-needed
2019-08-14 09:54:38 Edward Hope-Morley tags sts sts-sru-needed verification-needed verification-needed-bionic verification-queens-needed sts sts-sru-needed verification-done-bionic verification-needed verification-queens-needed
2019-08-20 15:41:40 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2019-08-20 15:41:59 Launchpad Janitor designate (Ubuntu Bionic): status Fix Committed Fix Released
2019-08-27 11:13:36 Edward Hope-Morley description [Impact] This patch is required to prevent pool-manager from creating unbounded amounts of status logs in the service_statuses table triggered by having > 1 log in there. [Test Case] * deploy openstack queens with designate * mysql> select count(*) from service_statuses where service_name="pool_manager"; should return 1 * try to add an extra entry: ts=date '+%Y-%m-%d %H:%M:%S' svc_host=`mysql -B -h$host -u${service} -p$passwd ${service} -e 'select hostname from service_statuses where service_name="pool_manager";'| tail -n 1` mysql -h$host -u${service} -p$passwd ${service} -e "insert into service_statuses values ('1234', '$ts', '$ts', 'pool_manager', '$svc_host', '$ts', 'UP', '{}', '{}');" * this should fail since the hostname/servicename columns should now be a unique contraint * can also check this with: mysql -h$host -u${service} -p$passwd ${service} -e "select * from INFORMATION_SCHEMA.TABLE_CONSTRAINTS where CONSTRAINT_TYPE='UNIQUE';"| grep service_statuses [Regression Potential] if the table already has multiple records for pool_manager in the service_statuses table, it will be necessary to (manually) delete all but one record in order for the upgrade to succeed. ---------------------------------------------------------------------------- Hi, The service_statuses table in Designate database is running full of records in our deployment: MariaDB [designate]> select count(*) from service_statuses; +----------+ | count(*) | +----------+ | 24474342 | +----------+ 1 row in set (7 min 19.09 sec) We got millions of rows in just couple of month. The problem is that the same services running on the same hosts create new record (instead of updating existing) during status report to Designate. This is how it looks in DB: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | 0dde2b5f228549d5995cb0338841bd50 | 2018-05-02 12:06:03 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:06:03 | UP | {} | {} | | 0e311d3000d8403d97066eba619490a3 | 2018-05-02 12:05:14 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:13 | UP | {} | {} | | 168448cd97cd428ea19318243570482c | 2018-05-02 12:05:48 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:48 | UP | {} | {} | | 1685d7f80d8c4f75b052680e5e2f40ae | 2018-05-02 12:05:59 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:58 | UP | {} | {} | | 192275eb33854b4091b981b0c32d04f7 | 2018-05-02 12:05:41 | NULL | worker | designate-worker-3446544-7fzqx | 2018-05-02 12:05:35 | UP | {} | {} | | 1e465011f21f47f096b54005675e8011 | 2018-05-02 12:05:25 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:05:25 | UP | {} | {} | | 22e0ab87b3cd4228bc191e49923d13ba | 2018-05-02 12:05:58 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:58 | UP | {} | {} | | 284d6b6bf53f47358fa14750d0c2a181 | 2018-05-02 12:05:18 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:18 | UP | {} | {} | | 3617ac740b9945c09f65f4b8cf0a72b5 | 2018-05-02 12:05:25 | NULL | api | designate-api-2042646259-drzws | 2018-05-02 12:05:25 | UP | {} | {} | | 3880157829f946398159f104b13a066b | 2018-05-02 12:06:00 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:06:00 | UP | {} | {} | The respective part of the code is: https://github.com/openstack/designate/blob/master/designate/central/service.py#L2879-L2898 It is not exactly clear to me why the id is needed to identify the service. The 'hostname' and 'service_name' seem to be enough - it is rather unlikely that there are two api services are running on the same host.. So, I've removed the id part:         if service_status.obj_attr_is_set('id'):             criterion["id"] = service_status.id and cleaned up the service_statuses table, now it looks fine: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | 1e89d10b0ddb459f8a9522ebd62fa629 | 2018-05-02 13:03:21 | 2018-05-02 13:03:37 | worker | designate-worker-2851226431-9274c | 2018-05-02 13:03:36 | UP | {} | {} | | 8d29d50270ee48999b78e90c1b88acf8 | 2018-05-02 13:03:23 | 2018-05-02 13:03:38 | api | designate-api-1699569558-9rvg6 | 2018-05-02 13:03:38 | UP | {} | {} | | 9602ccf55d204c7b86277e21af3d739f | 2018-05-02 13:03:18 | 2018-05-02 13:03:38 | producer | designate-producer-3892623035-6vwt0 | 2018-05-02 13:03:38 | UP | {} | {} | | cec705e082c542cdbfa83b9ed0c2b438 | 2018-05-02 13:03:19 | 2018-05-02 13:03:39 | api | designate-api-1699569558-r018m | 2018-05-02 13:03:39 | UP | {} | {} | | ddd48f13b67448d2949ced39d76e2958 | 2018-05-02 13:03:20 | 2018-05-02 13:03:40 | central | designate-central-2632925199-wwj7j | 2018-05-02 13:03:40 | UP | {} | {} | | ef0ae921a78d4ac090dc74a1dc686921 | 2018-05-02 13:03:21 | 2018-05-02 13:03:41 | mdns | designate-mdns-4162010419-pvclv | 2018-05-02 13:03:41 | UP | {} | {} | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ 6 rows in set (0.00 sec) Our Designate version is Pike, but the code seems to be the same in master branch. [Impact] This patch is required to prevent pool-manager from creating unbounded amounts of status logs in the service_statuses table triggered by having > 1 log in there. [Test Case] * deploy openstack queens with designate * mysql> select count(*) from service_statuses where service_name="pool_manager"; should return 1 * try to add an extra entry: ts=`date '+%Y-%m-%d %H:%M:%S'` svc_host=`mysql -B -h$host -u${service} -p$passwd ${service} -e 'select hostname from service_statuses where service_name="pool_manager";'| tail -n 1` mysql -h$host -u${service} -p$passwd ${service} -e "insert into service_statuses values ('1234', '$ts', '$ts', 'pool_manager', '$svc_host', '$ts', 'UP', '{}', '{}');" * this should fail since the hostname/servicename columns should now be a unique contraint * can also check this with: mysql -h$host -u${service} -p$passwd ${service} -e "select * from INFORMATION_SCHEMA.TABLE_CONSTRAINTS where CONSTRAINT_TYPE='UNIQUE';"| grep service_statuses [Regression Potential] if the table already has multiple records for pool_manager in the service_statuses table, it will be necessary to (manually) delete all but one record in order for the upgrade to succeed. ---------------------------------------------------------------------------- Hi, The service_statuses table in Designate database is running full of records in our deployment: MariaDB [designate]> select count(*) from service_statuses; +----------+ | count(*) | +----------+ | 24474342 | +----------+ 1 row in set (7 min 19.09 sec) We got millions of rows in just couple of month. The problem is that the same services running on the same hosts create new record (instead of updating existing) during status report to Designate. This is how it looks in DB: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+------------------------------------+---------------------+--------+-------+--------------+ | 0dde2b5f228549d5995cb0338841bd50 | 2018-05-02 12:06:03 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:06:03 | UP | {} | {} | | 0e311d3000d8403d97066eba619490a3 | 2018-05-02 12:05:14 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:13 | UP | {} | {} | | 168448cd97cd428ea19318243570482c | 2018-05-02 12:05:48 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:48 | UP | {} | {} | | 1685d7f80d8c4f75b052680e5e2f40ae | 2018-05-02 12:05:59 | NULL | api | designate-api-2042646259-6090v | 2018-05-02 12:05:58 | UP | {} | {} | | 192275eb33854b4091b981b0c32d04f7 | 2018-05-02 12:05:41 | NULL | worker | designate-worker-3446544-7fzqx | 2018-05-02 12:05:35 | UP | {} | {} | | 1e465011f21f47f096b54005675e8011 | 2018-05-02 12:05:25 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:05:25 | UP | {} | {} | | 22e0ab87b3cd4228bc191e49923d13ba | 2018-05-02 12:05:58 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:58 | UP | {} | {} | | 284d6b6bf53f47358fa14750d0c2a181 | 2018-05-02 12:05:18 | NULL | producer | designate-producer-855855776-cr8d9 | 2018-05-02 12:05:18 | UP | {} | {} | | 3617ac740b9945c09f65f4b8cf0a72b5 | 2018-05-02 12:05:25 | NULL | api | designate-api-2042646259-drzws | 2018-05-02 12:05:25 | UP | {} | {} | | 3880157829f946398159f104b13a066b | 2018-05-02 12:06:00 | NULL | mdns | designate-mdns-4198843580-lw6s2 | 2018-05-02 12:06:00 | UP | {} | {} | The respective part of the code is: https://github.com/openstack/designate/blob/master/designate/central/service.py#L2879-L2898 It is not exactly clear to me why the id is needed to identify the service. The 'hostname' and 'service_name' seem to be enough - it is rather unlikely that there are two api services are running on the same host.. So, I've removed the id part:         if service_status.obj_attr_is_set('id'):             criterion["id"] = service_status.id and cleaned up the service_statuses table, now it looks fine: MariaDB [designate]> select * from service_statuses; +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | id | created_at | updated_at | service_name | hostname | heartbeated_at | status | stats | capabilities | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ | 1e89d10b0ddb459f8a9522ebd62fa629 | 2018-05-02 13:03:21 | 2018-05-02 13:03:37 | worker | designate-worker-2851226431-9274c | 2018-05-02 13:03:36 | UP | {} | {} | | 8d29d50270ee48999b78e90c1b88acf8 | 2018-05-02 13:03:23 | 2018-05-02 13:03:38 | api | designate-api-1699569558-9rvg6 | 2018-05-02 13:03:38 | UP | {} | {} | | 9602ccf55d204c7b86277e21af3d739f | 2018-05-02 13:03:18 | 2018-05-02 13:03:38 | producer | designate-producer-3892623035-6vwt0 | 2018-05-02 13:03:38 | UP | {} | {} | | cec705e082c542cdbfa83b9ed0c2b438 | 2018-05-02 13:03:19 | 2018-05-02 13:03:39 | api | designate-api-1699569558-r018m | 2018-05-02 13:03:39 | UP | {} | {} | | ddd48f13b67448d2949ced39d76e2958 | 2018-05-02 13:03:20 | 2018-05-02 13:03:40 | central | designate-central-2632925199-wwj7j | 2018-05-02 13:03:40 | UP | {} | {} | | ef0ae921a78d4ac090dc74a1dc686921 | 2018-05-02 13:03:21 | 2018-05-02 13:03:41 | mdns | designate-mdns-4162010419-pvclv | 2018-05-02 13:03:41 | UP | {} | {} | +----------------------------------+---------------------+---------------------+--------------+-------------------------------------+---------------------+--------+-------+--------------+ 6 rows in set (0.00 sec) Our Designate version is Pike, but the code seems to be the same in master branch.
2019-08-27 11:16:21 Edward Hope-Morley tags sts sts-sru-needed verification-done-bionic verification-needed verification-queens-needed sts sts-sru-done verification-done verification-done-bionic verification-queens-done
2019-08-27 20:32:50 Corey Bryant cloud-archive/queens: status Fix Committed Fix Released