Activity log for bug #1649616

Date Who What changed Old value New value Message
2016-12-13 15:37:56 Alex Krzos bug added bug
2016-12-13 15:37:56 Alex Krzos attachment added Disk IO % util on Controller when Token Flush is running. https://bugs.launchpad.net/bugs/1649616/+attachment/4791197/+files/Token_flush-Disk_io.png
2016-12-22 03:05:54 Steve Martinelli keystone: importance Undecided Medium
2017-01-06 18:08:24 Alex Krzos attachment added RAID 1 disk pegged at end of token flush job. https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4800949/+files/Disk-io-on-raid1-token-flush.png
2017-02-24 22:54:57 Elvin Tubillara bug added subscriber Elvin Tubillara
2017-03-09 09:58:29 Peter Sabaini tags canonical-bootstack
2017-03-09 17:44:09 Peter Sabaini bug added subscriber The Canonical Sysadmins
2017-03-12 06:35:19 Dominique Poulain bug added subscriber Dominique Poulain
2017-04-06 21:09:59 OpenStack Infra keystone: status New In Progress
2017-04-06 21:09:59 OpenStack Infra keystone: assignee Peter Sabaini (peter-sabaini)
2017-04-12 06:59:27 Juan Antonio Osorio Robles bug task added puppet-keystone
2017-04-12 06:59:39 Juan Antonio Osorio Robles bug task added tripleo
2017-04-12 11:52:42 Juan Antonio Osorio Robles puppet-keystone: assignee Juan Antonio Osorio Robles (juan-osorio-robles)
2017-04-12 11:52:45 Juan Antonio Osorio Robles tripleo: assignee Juan Antonio Osorio Robles (juan-osorio-robles)
2017-04-13 01:23:00 Emilien Macchi tripleo: milestone pike-2
2017-04-13 01:23:02 Emilien Macchi puppet-keystone: status New Triaged
2017-04-13 01:23:06 Emilien Macchi tripleo: status New Triaged
2017-04-13 01:23:08 Emilien Macchi puppet-keystone: importance Undecided Medium
2017-04-13 01:23:11 Emilien Macchi tripleo: importance Undecided Medium
2017-04-19 15:03:15 OpenStack Infra keystone: status In Progress Fix Released
2017-04-19 15:14:02 OpenStack Infra tags canonical-bootstack canonical-bootstack in-stable-ocata
2017-04-26 21:33:29 Drew Freiberger merge proposal linked https://code.launchpad.net/~canonical-bootstack/bootstack-ops/+git/bootstack-ops/+merge/323253
2017-05-29 17:13:22 Jorge Niedbalski bug task added cloud-archive
2017-05-29 17:13:37 Jorge Niedbalski tags canonical-bootstack in-stable-ocata canonical-bootstack in-stable-ocata sts
2017-05-29 17:15:16 Jorge Niedbalski bug task added keystone (Ubuntu)
2017-05-29 17:15:39 Jorge Niedbalski nominated for series Ubuntu Yakkety
2017-05-29 17:15:39 Jorge Niedbalski nominated for series Ubuntu Zesty
2017-05-29 17:15:39 Jorge Niedbalski nominated for series Ubuntu Xenial
2017-05-29 17:23:17 Eric Desrochers bug task added keystone (Ubuntu Xenial)
2017-05-29 17:23:27 Eric Desrochers bug task added keystone (Ubuntu Yakkety)
2017-05-29 17:23:40 Eric Desrochers bug task added keystone (Ubuntu Zesty)
2017-05-30 11:59:13 Corey Bryant keystone (Ubuntu Xenial): importance Undecided Medium
2017-05-30 11:59:18 Corey Bryant keystone (Ubuntu Yakkety): importance Undecided Medium
2017-05-30 11:59:24 Corey Bryant keystone (Ubuntu Zesty): importance Undecided Medium
2017-05-30 12:01:34 Corey Bryant keystone (Ubuntu Xenial): status New Triaged
2017-05-30 12:01:38 Corey Bryant keystone (Ubuntu Yakkety): status New Triaged
2017-05-30 12:01:46 Corey Bryant keystone (Ubuntu Zesty): status New Triaged
2017-05-30 12:03:00 Corey Bryant nominated for series cloud-archive/mitaka
2017-05-30 12:03:00 Corey Bryant bug task added cloud-archive/mitaka
2017-05-30 12:03:00 Corey Bryant nominated for series cloud-archive/newton
2017-05-30 12:03:00 Corey Bryant bug task added cloud-archive/newton
2017-05-30 12:03:00 Corey Bryant nominated for series cloud-archive/ocata
2017-05-30 12:03:00 Corey Bryant bug task added cloud-archive/ocata
2017-05-30 12:03:18 Corey Bryant cloud-archive/mitaka: importance Undecided Medium
2017-05-30 12:03:18 Corey Bryant cloud-archive/mitaka: status New Triaged
2017-05-30 12:04:06 Corey Bryant cloud-archive/newton: importance Undecided Medium
2017-05-30 12:04:06 Corey Bryant cloud-archive/newton: status New Triaged
2017-05-30 12:04:25 Corey Bryant cloud-archive/ocata: importance Undecided Medium
2017-05-30 12:04:25 Corey Bryant cloud-archive/ocata: status New Triaged
2017-05-31 15:22:06 Launchpad Janitor merge proposal linked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324870
2017-05-31 15:23:07 Launchpad Janitor merge proposal linked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324871
2017-05-31 15:23:40 Launchpad Janitor merge proposal linked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324872
2017-05-31 15:25:59 Launchpad Janitor merge proposal linked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324873
2017-05-31 15:26:28 Jorge Niedbalski cloud-archive: status New In Progress
2017-05-31 15:26:28 Jorge Niedbalski cloud-archive: assignee Jorge Niedbalski (niedbalski)
2017-05-31 15:26:46 Jorge Niedbalski cloud-archive/mitaka: status Triaged In Progress
2017-05-31 15:26:46 Jorge Niedbalski cloud-archive/mitaka: assignee Jorge Niedbalski (niedbalski)
2017-05-31 15:27:02 Jorge Niedbalski cloud-archive/newton: status Triaged In Progress
2017-05-31 15:27:02 Jorge Niedbalski cloud-archive/newton: assignee Jorge Niedbalski (niedbalski)
2017-05-31 15:27:24 Jorge Niedbalski cloud-archive/ocata: status Triaged In Progress
2017-05-31 15:27:24 Jorge Niedbalski cloud-archive/ocata: assignee Jorge Niedbalski (niedbalski)
2017-05-31 15:27:45 Jorge Niedbalski keystone (Ubuntu): importance Undecided High
2017-05-31 15:27:45 Jorge Niedbalski keystone (Ubuntu): status New In Progress
2017-05-31 15:27:45 Jorge Niedbalski keystone (Ubuntu): assignee Jorge Niedbalski (niedbalski)
2017-05-31 15:28:03 Jorge Niedbalski keystone (Ubuntu Xenial): status Triaged In Progress
2017-05-31 15:28:03 Jorge Niedbalski keystone (Ubuntu Xenial): assignee Jorge Niedbalski (niedbalski)
2017-05-31 15:28:21 Jorge Niedbalski keystone (Ubuntu Yakkety): importance Medium High
2017-05-31 15:28:21 Jorge Niedbalski keystone (Ubuntu Yakkety): status Triaged In Progress
2017-05-31 15:28:21 Jorge Niedbalski keystone (Ubuntu Yakkety): assignee Jorge Niedbalski (niedbalski)
2017-05-31 15:28:51 Jorge Niedbalski keystone (Ubuntu Zesty): importance Medium High
2017-05-31 15:28:51 Jorge Niedbalski keystone (Ubuntu Zesty): status Triaged In Progress
2017-05-31 15:28:51 Jorge Niedbalski keystone (Ubuntu Zesty): assignee Jorge Niedbalski (niedbalski)
2017-05-31 20:42:22 Jorge Niedbalski attachment added fix-lp-1649616-artful.debdiff https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886957/+files/fix-lp-1649616-artful.debdiff
2017-05-31 20:43:07 Jorge Niedbalski attachment added fix-lp-1649616-zesty.debdiff https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886958/+files/fix-lp-1649616-zesty.debdiff
2017-05-31 20:43:33 Jorge Niedbalski attachment added fix-lp-1649616-yakkety.debdiff https://bugs.launchpad.net/keystone/+bug/1649616/+attachment/4886959/+files/fix-lp-1649616-yakkety.debdiff
2017-05-31 20:43:59 Jorge Niedbalski keystone (Ubuntu Xenial): importance Medium High
2017-05-31 20:44:32 Jorge Niedbalski attachment added fix-lp-1649616-xenial.debdiff https://bugs.launchpad.net/ubuntu/xenial/+source/keystone/+bug/1649616/+attachment/4886960/+files/fix-lp-1649616-xenial.debdiff
2017-05-31 20:44:52 Jorge Niedbalski bug added subscriber Ubuntu Sponsors Team
2017-05-31 21:02:14 Eric Desrochers tags canonical-bootstack in-stable-ocata sts canonical-bootstack in-stable-ocata sts sts-sru-needed
2017-05-31 21:02:20 Eric Desrochers bug added subscriber Eric Desrochers
2017-06-05 16:17:52 Jorge Niedbalski description The Keystone token flush job can get into a state where it will never complete because the transaction size exceeds the mysql galara transaction size - wsrep_max_ws_size (1073741824). Steps to Reproduce: 1. Authenticate many times 2. Observe that keystone token flush job runs (should be a very long time depending on disk) >20 hours in my environment 3. Observe errors in mysql.log indicating a transaction that is too large Actual results: Expired tokens are not actually flushed from the database without any errors in keystone.log. Only errors appear in mysql.log. Expected results: Expired tokens to be removed from the database Additional info: It is likely that you can demonstrate this with less than 1 million tokens as the >1 million token table is larger than 13GiB and the max transaction size is 1GiB, my token bench-marking Browbeat job creates more than needed. Once the token flush job can not complete the token table will never decrease in size and eventually the cloud will run out of disk space. Furthermore the flush job will consume disk utilization resources. This was demonstrated on slow disks (Single 7.2K SATA disk). On faster disks you will have more capacity to generate tokens, you can then generate the number of tokens to exceed the transaction size even faster. Log evidence: [root@overcloud-controller-0 log]# grep " Total expired" /var/log/keystone/keystone.log 2016-12-08 01:33:40.530 21614 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1082434 2016-12-09 09:31:25.301 14120 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1084241 2016-12-11 01:35:39.082 4223 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1086504 2016-12-12 01:08:16.170 32575 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1087823 2016-12-13 01:22:18.121 28669 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1089202 [root@overcloud-controller-0 log]# tail mysqld.log 161208 1:33:41 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161208 1:33:41 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161209 9:31:26 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161209 9:31:26 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161211 1:35:39 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161211 1:35:40 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161212 1:08:16 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161212 1:08:17 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161213 1:22:18 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161213 1:22:19 [ERROR] WSREP: rbr write fail, data_len: 0, 2 Disk utilization issue graph is attached. The entire job in that graph takes from the first spike is disk util(~5:18UTC) and culminates in about ~90 minutes of pegging the disk (between 1:09utc to 2:43utc). [Impact] * The Keystone token flush job can get into a state where it will never complete because the transaction size exceeds the mysql galara transaction size - wsrep_max_ws_size (1073741824). [Test Case] 1. Authenticate many times 2. Observe that keystone token flush job runs (should be a very long time depending on disk) >20 hours in my environment 3. Observe errors in mysql.log indicating a transaction that is too large Actual results: Expired tokens are not actually flushed from the database without any errors in keystone.log. Only errors appear in mysql.log. Expected results: Expired tokens to be removed from the database [Additional info:] It is likely that you can demonstrate this with less than 1 million tokens as the >1 million token table is larger than 13GiB and the max transaction size is 1GiB, my token bench-marking Browbeat job creates more than needed. Once the token flush job can not complete the token table will never decrease in size and eventually the cloud will run out of disk space. Furthermore the flush job will consume disk utilization resources. This was demonstrated on slow disks (Single 7.2K SATA disk). On faster disks you will have more capacity to generate tokens, you can then generate the number of tokens to exceed the transaction size even faster. Log evidence: [root@overcloud-controller-0 log]# grep " Total expired" /var/log/keystone/keystone.log 2016-12-08 01:33:40.530 21614 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1082434 2016-12-09 09:31:25.301 14120 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1084241 2016-12-11 01:35:39.082 4223 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1086504 2016-12-12 01:08:16.170 32575 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1087823 2016-12-13 01:22:18.121 28669 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1089202 [root@overcloud-controller-0 log]# tail mysqld.log 161208 1:33:41 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161208 1:33:41 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161209 9:31:26 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161209 9:31:26 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161211 1:35:39 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161211 1:35:40 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161212 1:08:16 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161212 1:08:17 [ERROR] WSREP: rbr write fail, data_len: 0, 2 161213 1:22:18 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073774592 161213 1:22:19 [ERROR] WSREP: rbr write fail, data_len: 0, 2 Disk utilization issue graph is attached. The entire job in that graph takes from the first spike is disk util(~5:18UTC) and culminates in about ~90 minutes of pegging the disk (between 1:09utc to 2:43utc). [Regression Potential] * Not identified
2017-06-05 20:06:48 Lance Bragstad nominated for series keystone/newton
2017-06-05 20:06:48 Lance Bragstad bug task added keystone/newton
2017-06-05 20:06:48 Lance Bragstad nominated for series keystone/ocata
2017-06-05 20:06:48 Lance Bragstad bug task added keystone/ocata
2017-06-05 20:09:54 Lance Bragstad keystone/newton: importance Undecided Medium
2017-06-05 20:09:54 Lance Bragstad keystone/newton: status New In Progress
2017-06-05 20:10:14 Lance Bragstad keystone/ocata: importance Undecided Medium
2017-06-05 20:10:14 Lance Bragstad keystone/ocata: status New In Progress
2017-06-07 09:15:01 James Page removed subscriber Ubuntu Sponsors Team
2017-06-07 09:15:11 James Page bug added subscriber Ubuntu Stable Release Updates Team
2017-06-08 01:31:57 Eric Desrochers tags canonical-bootstack in-stable-ocata sts sts-sru-needed canonical-bootstack in-stable-ocata sts sts-sru-done
2017-06-08 11:42:25 Eric Desrochers bug added subscriber SRU Verification
2017-06-08 11:42:43 Edward Hope-Morley tags canonical-bootstack in-stable-ocata sts sts-sru-done canonical-bootstack in-stable-ocata sts sts-sru-needed
2017-06-08 11:44:26 Eric Desrochers tags canonical-bootstack in-stable-ocata sts sts-sru-needed canonical-bootstack in-stable-ocata sts sts-sru-needd
2017-06-08 11:44:34 Eric Desrochers tags canonical-bootstack in-stable-ocata sts sts-sru-needd canonical-bootstack in-stable-ocata sts sts-sru-needed
2017-06-08 20:53:44 Emilien Macchi tripleo: milestone pike-2 pike-3
2017-06-16 10:15:14 Andy Whitcroft keystone (Ubuntu Zesty): status In Progress Fix Committed
2017-06-16 10:15:20 Andy Whitcroft tags canonical-bootstack in-stable-ocata sts sts-sru-needed canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed
2017-06-19 07:49:45 James Page cloud-archive/ocata: status In Progress Fix Committed
2017-06-19 07:49:48 James Page tags canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed verification-ocata-needed
2017-06-22 00:32:44 Graeme Gillies bug added subscriber Graeme Gillies
2017-07-03 12:28:12 Andy Whitcroft keystone (Ubuntu Yakkety): status In Progress Fix Committed
2017-07-03 12:28:21 Andy Whitcroft tags canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed verification-ocata-needed canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed verification-needed-yakkety verification-ocata-needed
2017-07-03 12:29:13 Andy Whitcroft keystone (Ubuntu Xenial): status In Progress Fix Committed
2017-07-03 12:29:21 Andy Whitcroft tags canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed verification-needed-yakkety verification-ocata-needed canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-ocata-needed
2017-07-04 14:03:32 James Page cloud-archive/mitaka: status In Progress Fix Committed
2017-07-04 14:03:35 James Page tags canonical-bootstack in-stable-ocata sts sts-sru-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-ocata-needed canonical-bootstack in-stable-ocata sts sts-sru-needed verification-mitaka-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-ocata-needed
2017-07-04 14:04:26 James Page cloud-archive/newton: status In Progress Fix Committed
2017-07-04 14:04:28 James Page tags canonical-bootstack in-stable-ocata sts sts-sru-needed verification-mitaka-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-ocata-needed canonical-bootstack in-stable-ocata sts sts-sru-needed verification-mitaka-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-newton-needed verification-ocata-needed
2017-07-05 19:01:51 Emilien Macchi tripleo: status Triaged In Progress
2017-07-11 15:26:31 OpenStack Infra keystone/ocata: assignee Raildo Mascena de Sousa Filho (raildo)
2017-07-11 15:38:12 OpenStack Infra keystone/newton: assignee Raildo Mascena de Sousa Filho (raildo)
2017-07-12 07:41:45 OpenStack Infra tags canonical-bootstack in-stable-ocata sts sts-sru-needed verification-mitaka-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-newton-needed verification-ocata-needed canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-mitaka-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-newton-needed verification-ocata-needed
2017-07-13 16:17:57 Jorge Niedbalski tags canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-mitaka-needed verification-needed verification-needed-xenial verification-needed-yakkety verification-newton-needed verification-ocata-needed canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done-xenial verification-mitaka-done verification-needed verification-needed-yakkety verification-newton-needed verification-ocata-needed
2017-07-13 16:20:43 Jorge Niedbalski merge proposal unlinked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324873
2017-07-13 16:21:55 Jorge Niedbalski merge proposal unlinked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324872
2017-07-13 17:05:41 Brian Murray tags canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done-xenial verification-mitaka-done verification-needed verification-needed-yakkety verification-newton-needed verification-ocata-needed canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done-xenial verification-mitaka-done verification-needed verification-needed-yakkety verification-needed-zesty verification-newton-needed verification-ocata-needed
2017-07-13 21:48:33 Jorge Niedbalski tags canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done-xenial verification-mitaka-done verification-needed verification-needed-yakkety verification-needed-zesty verification-newton-needed verification-ocata-needed canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done-xenial verification-done-yakkety verification-mitaka-done verification-needed verification-needed-zesty verification-newton-done verification-ocata-needed
2017-07-13 21:48:46 Jorge Niedbalski merge proposal unlinked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324871
2017-07-13 21:49:07 Jorge Niedbalski merge proposal unlinked https://code.launchpad.net/~niedbalski/ubuntu/+source/keystone/+git/keystone/+merge/324870
2017-07-13 22:22:42 Jorge Niedbalski tags canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done-xenial verification-done-yakkety verification-mitaka-done verification-needed verification-needed-zesty verification-newton-done verification-ocata-needed canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done verification-done-xenial verification-done-yakkety verification-done-zesty verification-mitaka-done verification-newton-done verification-ocata-done
2017-07-17 14:43:45 Ɓukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2017-07-17 14:53:48 Launchpad Janitor keystone (Ubuntu Zesty): status Fix Committed Fix Released
2017-07-17 15:20:03 Launchpad Janitor keystone (Ubuntu Yakkety): status Fix Committed Fix Released
2017-07-17 15:23:42 Launchpad Janitor keystone (Ubuntu Xenial): status Fix Committed Fix Released
2017-07-18 11:52:47 James Page cloud-archive/ocata: status Fix Committed Fix Released
2017-07-25 11:02:15 James Page cloud-archive/mitaka: status Fix Committed Fix Released
2017-07-25 11:03:04 James Page cloud-archive/newton: status Fix Committed Fix Released
2017-07-30 05:02:54 Emilien Macchi tripleo: milestone pike-3 pike-rc1
2017-07-31 15:55:58 Emilien Macchi tripleo: milestone pike-rc1 queens-1
2017-07-31 16:13:39 Juan Antonio Osorio Robles tripleo: status In Progress Fix Released
2017-08-07 10:45:34 Edward Hope-Morley tags canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-needed verification-done verification-done-xenial verification-done-yakkety verification-done-zesty verification-mitaka-done verification-newton-done verification-ocata-done canonical-bootstack in-stable-newton in-stable-ocata sts sts-sru-done verification-done verification-done-xenial verification-done-yakkety verification-done-zesty verification-mitaka-done verification-newton-done verification-ocata-done
2017-08-31 19:07:02 Corey Bryant keystone (Ubuntu): status In Progress Invalid
2017-08-31 19:07:19 Corey Bryant cloud-archive: status In Progress Invalid
2018-04-04 18:40:42 Tobias Urdin puppet-keystone: status Triaged Fix Released
2018-10-24 17:55:11 Morgan Fainberg keystone/newton: status In Progress Won't Fix
2018-10-24 17:56:34 Morgan Fainberg keystone/ocata: status In Progress Fix Released