Bug #1649616 “Keystone Token Flush job does not complete in HA d...” : Series newton : Bugs : OpenStack Identity (keystone)

Revision history for this message

Alex Krzos (akrzos) wrote on 2016-12-13:

#1

Disk IO % util on Controller when Token Flush is running. Edit (63.7 KiB, image/png)

Revision history for this message

Steve Martinelli (stevemar) wrote on 2016-12-15:

#2

Hi Alex,

So someone finally hit bug https://bugs.launchpad.net/keystone/+bug/1609511 in a production environment.

A few questions so I gauge the severity properly.

- Is it possible to run the cron job that flushes tokens more frequently?

- What is preventing you from moving to fernet tokens? They are not persisted.

Lastly, Sam Leong had a patch for this issue, it's available here: https://review.openstack.org/#/c/351428/

Unfortunately, Sam has stopped working on keystone these days.

Revision history for this message

Alex Krzos (akrzos) wrote on 2016-12-15:

#3

Hi Steve,

Actually this was not on a production cloud but a performance testing cloud. This bug is likely a duplicate to the one you linked me over to. One major difference is there are no errors in the keystone log file as the previous bug noted though. (but there is still an error in mysql log file)

To answer your questions though (in the context of performance testing)

- Is it possible to run the cron job that flushes tokens more frequently?

I suppose we can do this, to benchmark more "deterministically" we should probably clean the token table out prior to each iteration (357 iterations during this test)

- What is preventing you from moving to fernet tokens? They are not persisted.

I am actually running the same benchmarks with Fernet tokens as well.

As you already pointed out there is several options to address this:
* Increase frequency of token flushing (at expense of the resource consumption demanded by token flushes)
* Switch to Fernet tokens (No need for a flush, but different performance characteristics)
* Increase size of allowed mysql transaction (increase the size of the flush)

Depending upon the workload in the cloud, some options might be better I suppose.

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2016-12-20:

#4

Alex - is the only difference between this report and https://bugs.launchpad.net/keystone/+bug/1609511 the fact the 1609511 has a more detailed error in the logs? What logging level was set when you ran into this issue?

Steve Martinelli (stevemar) on 2016-12-22

Changed in keystone:
importance:	Undecided → Medium

Revision history for this message

Samuel de Medeiros Queiroz (samueldmq) wrote on 2017-01-03:

#5

What should someone do in the case this is hit in production ?

Fernet could be used, but the migration can take some time. What to do in the meantime ?

On the other hand, nobody has hit this in production yet (at least not reported publicly).
Should we mark this as won't fix for now rather than working in a fix that is not useful for anyone now ?

Revision history for this message

Alex Krzos (akrzos) wrote on 2017-01-03:

#6

Lance,

I reviewed the log from the other bug (1609511) another question here is how come with that cloud the error was logged into keystone.log vs my cloud the error showed up under mysql but was silent in keystone.log. I will re-run on another setup I have to see if this is still the case or if I just missed the fact keystone does log it or something is configured to silence that issue.

Revision history for this message

Alex Krzos (akrzos) wrote on 2017-01-06:

#7

Download full text (9.3 KiB)

I reran to confirm on a similar setup but with a slightly better disk layout (RAID 1 Mirror rather than single SATA disk) and the results are the same. I reviewed the Keystone log files and did find the same stack trace as the other bug. So this is logged correctly with the deployment that I am using.

Aside from the tokens never clearing and the job running for almost 24 hours, when this job runs and doesn't complete the cloud still pays the same penalty in disk utilization attempting to remove the expired tokens(Every day forth).

Here is the log output related to the unsuccessful token flush and I will attach another disk io % util on the controller that the database processes are handling the request from the token flush process.

2017-01-06 03:43:05.810 1013422 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1003291 [25/213]
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1180, u'Got error 5 during COMMIT')
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters self.engine.dialect.do_commit(self.connection)
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters dbapi_connection.commit()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 719, in commit
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters self._read_ok_packet()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 698, in _read_ok_packet
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters pkt = self._read_packet()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo)
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters File "...

I reran to confirm on a similar setup but with a slightly better disk layout (RAID 1 Mirror rather than single SATA disk) and the results are the same.  I reviewed the Keystone log files and did find the same stack trace as the other bug.  So this is logged correctly with the deployment that I am using.

Aside from the tokens never clearing and the job running for almost 24 hours, when this job runs and doesn't complete the cloud still pays the same penalty in disk utilization attempting to remove the expired tokens(Every day forth).

Here is the log output related to the unsuccessful token flush and I will attach another disk io % util on the controller that the database processes are handling the request from the token flush process.

2017-01-06 03:43:05.810 1013422 INFO keystone.token.persistence.backends.sql [-] Total expired tokens removed: 1003291                                                                                                           [25/213]
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1180, u'Got error 5 during COMMIT')
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     self.engine.dialect.do_commit(self.connection)
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     dbapi_connection.commit()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 719, in commit
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     self._read_ok_packet()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 698, in _read_ok_packet
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     pkt = self._read_packet()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     _check_mysql_exception(errinfo)
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters     raise InternalError(errno, errorvalue)
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1180, u'Got error 5 during COMMIT')
2017-01-06 04:06:23.540 1013422 ERROR oslo_db.sqlalchemy.exc_filters 
2017-01-06 04:06:23.581 1013422 CRITICAL keystone [-] DBError: (pymysql.err.InternalError) (1180, u'Got error 5 during COMMIT')
2017-01-06 04:06:23.581 1013422 ERROR keystone Traceback (most recent call last):
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/bin/keystone-manage", line 10, in <module>
2017-01-06 04:06:23.581 1013422 ERROR keystone     sys.exit(main())
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/manage.py", line 43, in main
2017-01-06 04:06:23.581 1013422 ERROR keystone     cli.main(argv=sys.argv, config_files=config_files)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1256, in main
2017-01-06 04:06:23.581 1013422 ERROR keystone     CONF.command.cmd_class.main()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 767, in main
2017-01-06 04:06:23.581 1013422 ERROR keystone     token_manager.flush_expired_tokens()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/token/persistence/backends/sql.py", line 286, in flush_expired_tokens
2017-01-06 04:06:23.581 1013422 ERROR keystone     LOG.info(_LI('Total expired tokens removed: %d'), total_removed)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-01-06 04:06:23.581 1013422 ERROR keystone     self.gen.next()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 875, in _transaction_scope
2017-01-06 04:06:23.581 1013422 ERROR keystone     yield resource
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-01-06 04:06:23.581 1013422 ERROR keystone     self.gen.next()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 522, in _session
2017-01-06 04:06:23.581 1013422 ERROR keystone     self._end_session_transaction(self.session)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 543, in _end_session_transaction
2017-01-06 04:06:23.581 1013422 ERROR keystone     session.commit()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in commit
2017-01-06 04:06:23.581 1013422 ERROR keystone     self.transaction.commit()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in commit
2017-01-06 04:06:23.581 1013422 ERROR keystone     t[1].commit()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in commit
2017-01-06 04:06:23.581 1013422 ERROR keystone     self._do_commit()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in _do_commit
2017-01-06 04:06:23.581 1013422 ERROR keystone     self.connection._commit_impl()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in _commit_impl
2017-01-06 04:06:23.581 1013422 ERROR keystone     self._handle_dbapi_exception(e, None, None, None, None)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2017-01-06 04:06:23.581 1013422 ERROR keystone     util.raise_from_cause(newraise, exc_info)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2017-01-06 04:06:23.581 1013422 ERROR keystone     reraise(type(exception), exception, tb=exc_tb)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
2017-01-06 04:06:23.581 1013422 ERROR keystone     self.engine.dialect.do_commit(self.connection)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
2017-01-06 04:06:23.581 1013422 ERROR keystone     dbapi_connection.commit()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 719, in commit
2017-01-06 04:06:23.581 1013422 ERROR keystone     self._read_ok_packet()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 698, in _read_ok_packet
2017-01-06 04:06:23.581 1013422 ERROR keystone     pkt = self._read_packet()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2017-01-06 04:06:23.581 1013422 ERROR keystone     packet.check_error()
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2017-01-06 04:06:23.581 1013422 ERROR keystone     err.raise_mysql_exception(self._data)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-01-06 04:06:23.581 1013422 ERROR keystone     _check_mysql_exception(errinfo)
2017-01-06 04:06:23.581 1013422 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2017-01-06 04:06:23.581 1013422 ERROR keystone     raise InternalError(errno, errorvalue)
2017-01-06 04:06:23.581 1013422 ERROR keystone DBError: (pymysql.err.InternalError) (1180, u'Got error 5 during COMMIT')
2017-01-06 04:06:23.581 1013422 ERROR keystone

Revision history for this message

Alex Krzos (akrzos) wrote on 2017-01-06:

#8

RAID 1 disk pegged at end of token flush job. Edit (52.0 KiB, image/png)

Do note that this is at the "end" of the token flush job. The actual job ran for much longer, but this end part pegs the disk out attempting to commit.

Peter Sabaini (peter-sabaini) on 2017-03-09

tags:

added: canonical-bootstack

Revision history for this message

Peter Sabaini (peter-sabaini) wrote on 2017-03-09:

#9

We seem to be hitting this in production.

Mysql log has:

170309 9:06:23 [Warning] WSREP: transaction size limit (1073774592) exceeded: 1073807360
170309 9:06:23 [ERROR] WSREP: rbr write fail, data_len: 0, 2

The leftover GRA log files contain only DELETE statements for keystone.token afaict.

Keystone log contains:

(oslo_db.sqlalchemy.exc_filters): 2017-03-09 09:07:06,576 ERROR DBAPIError exception wrapped from (pymysql.err.InternalError) (1180, u'Got error 5 during COMMIT')
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
    dbapi_connection.commit()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 758, in commit
    self._read_ok_packet()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 737, in _read_ok_packet
    pkt = self._read_packet()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
    packet.check_error()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
InternalError: (1180, u'Got error 5 during COMMIT')

The token table has ~500k entries.

Revision history for this message

Lance Bragstad (lbragstad) wrote on 2017-03-10:

#10

What if it were possible to give `keystone-manage token_flush` a batch size as an argument? This will give deployments with massive amounts of expired token the ability to clean them out in smaller batches.

It's certainly more intensive for the operator and really only makes sense for recovery situations like this. Thoughts?

Revision history for this message

Peter Sabaini (peter-sabaini) wrote on 2017-04-06:

#11

Lance, the token_flush batches fine, this has been introduced with the fix to bug #1188378.

The problem afaict is just that this is done in a single transaction, with the result that this again can break replication.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-06: Fix proposed to keystone (master)

#12

Fix proposed to branch: master
Review: https://review.openstack.org/454351

Changed in keystone:
assignee:	nobody → Peter Sabaini (peter-sabaini)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-12: Related fix proposed to puppet-keystone (master)

#13

Related fix proposed to branch: master
Review: https://review.openstack.org/456086

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-12: Related fix proposed to tripleo-heat-templates (master)

#14

Related fix proposed to branch: master
Review: https://review.openstack.org/456182

Juan Antonio Osorio Robles (juan-osorio-robles) on 2017-04-12

Changed in puppet-keystone:
assignee:	nobody → Juan Antonio Osorio Robles (juan-osorio-robles)
Changed in tripleo:
assignee:	nobody → Juan Antonio Osorio Robles (juan-osorio-robles)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-12: Related fix merged to puppet-keystone (master)

#15

Reviewed: https://review.openstack.org/456086
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=bce06b24ac9439329fd8973a477f1ff9aca7be15
Submitter: Jenkins
Branch: master

commit bce06b24ac9439329fd8973a477f1ff9aca7be15
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Wed Apr 12 10:09:28 2017 +0300

Run token flush cron job twice a day

Running the token flush once a day is seldom for larger deployments.
This changes the frequency to be twice a day instead of once.

Change-Id: Ia0b0fb422318712f4b0f4d023cbb3a61d40bb85d
Related-Bug: #1649616

Emilien Macchi (emilienm) on 2017-04-13

Changed in tripleo:
milestone:	none → pike-2
Changed in puppet-keystone:
status:	New → Triaged
Changed in tripleo:
status:	New → Triaged
Changed in puppet-keystone:
importance:	Undecided → Medium
Changed in tripleo:
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-18: Related fix proposed to puppet-keystone (master)

#16

Related fix proposed to branch: master
Review: https://review.openstack.org/457553

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-18: Related fix merged to tripleo-heat-templates (master)

#17

Reviewed: https://review.openstack.org/456182
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=65e643aca2202f031db94f1ccd3d44e195e5e772
Submitter: Jenkins
Branch: master

commit 65e643aca2202f031db94f1ccd3d44e195e5e772
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Wed Apr 12 14:31:53 2017 +0300

Run token flush cron job hourly by default

    Running this job once a day has proven problematic for large
    deployments as seen in the bug report. Setting it to run hourly
    would be an improvement to the current situation, as the flushes
    wouldn't need to process as much data.

Note that this only affects people using UUID as the token provider.

Change-Id: I462e4da2bfdbcba0403ecde5d613386938e2283a
Related-Bug: #1649616

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-19: Related fix merged to puppet-keystone (master)

#18

Reviewed: https://review.openstack.org/457553
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=f694b5551f896042df6aeb751c65986ef3342f54
Submitter: Jenkins
Branch: master

commit f694b5551f896042df6aeb751c65986ef3342f54
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Apr 18 13:13:27 2017 +0300

Change keystone token flush to run hourly

    In a recent commit [1] the keystone token flush cron job was changed to
    run twice a day. However, this change was not enough for big
    deployments.

    After getting some customer feedback and looking at what other projects
    are doing [2] [3] [4]. It seems that running this job hourly is the way
    to go.

    [1] Ia0b0fb422318712f4b0f4d023cbb3a61d40bb85d
    [2] https://www.ibm.com/support/knowledgecenter/en/SSB27U_6.4.0/com.ibm.zvm.v640.hcpo4/exptoken.htm
    [3] https://review.openstack.org/#/c/88670/8
    [4] https://github.com/openstack/charm-keystone/blob/master/templates/keystone-token-flush

Change-Id: I6ec7ec8111bd93e5638cfe96189e36f0e0691d65
Related-Bug: #1649616

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-19: Related fix proposed to puppet-keystone (stable/ocata)

#19

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/458002

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-19: Related fix proposed to tripleo-heat-templates (stable/ocata)

#20

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/458005

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-19: Fix merged to keystone (master)

#21

Reviewed: https://review.openstack.org/454351
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=dc7f81083180eeb5233f7007e3d2514cc0d7c6d3
Submitter: Jenkins
Branch: master

commit dc7f81083180eeb5233f7007e3d2514cc0d7c6d3
Author: Peter Sabaini <email address hidden>
Date: Thu Apr 6 23:06:29 2017 +0200

Make flushing tokens more robust

Commit token flushes between batches in order to lower resource
consumption and make flushing more robust for replication

Change-Id: I9be37e420353a336a8acd820eadd47d4bcf7324f
Closes-Bug: #1649616

Changed in keystone:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-19: Related fix merged to puppet-keystone (stable/ocata)

#22

Reviewed: https://review.openstack.org/458002
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=90ffc7f6008370e9d9893ab69f31683352c854c3
Submitter: Jenkins
Branch: stable/ocata

commit 90ffc7f6008370e9d9893ab69f31683352c854c3
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Apr 18 13:13:27 2017 +0300

Change keystone token flush to run hourly

    In a recent commit [1] the keystone token flush cron job was changed to
    run twice a day. However, this change was not enough for big
    deployments.

    After getting some customer feedback and looking at what other projects
    are doing [2] [3] [4]. It seems that running this job hourly is the way
    to go.

    [1] Ia0b0fb422318712f4b0f4d023cbb3a61d40bb85d
    [2] https://www.ibm.com/support/knowledgecenter/en/SSB27U_6.4.0/com.ibm.zvm.v640.hcpo4/exptoken.htm
    [3] https://review.openstack.org/#/c/88670/8
    [4] https://github.com/openstack/charm-keystone/blob/master/templates/keystone-token-flush

    Conflicts:
     manifests/cron/token_flush.pp
     spec/acceptance/keystone_federation_identity_provider_spec.rb
     spec/acceptance/keystone_federation_shibboleth_spec.rb
     spec/acceptance/keystone_wsgi_apache_spec.rb
     spec/classes/keystone_cron_token_flush_spec.rb

    (cherry picked from commit f694b5551f896042df6aeb751c65986ef3342f54)
    Change-Id: I6ec7ec8111bd93e5638cfe96189e36f0e0691d65
    Related-Bug: #1649616

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-21: Related fix merged to tripleo-heat-templates (stable/ocata)

#23

Reviewed: https://review.openstack.org/458005
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c1fc74c0f3a8ba34032ac40ee67ef3bc4b7c9d3e
Submitter: Jenkins
Branch: stable/ocata

commit c1fc74c0f3a8ba34032ac40ee67ef3bc4b7c9d3e
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Wed Apr 12 14:31:53 2017 +0300

Run token flush cron job hourly by default

    Running this job once a day has proven problematic for large
    deployments as seen in the bug report. Setting it to run hourly
    would be an improvement to the current situation, as the flushes
    wouldn't need to process as much data.

Note that this only affects people using UUID as the token provider.

    Change-Id: I462e4da2bfdbcba0403ecde5d613386938e2283a
    Related-Bug: #1649616
    (cherry picked from commit 65e643aca2202f031db94f1ccd3d44e195e5e772)

Jorge Niedbalski (niedbalski) on 2017-05-29

tags:

added: sts

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-30: Fix proposed to keystone (stable/ocata)

#24

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/469069

Corey Bryant (corey.bryant) on 2017-05-30

Changed in keystone (Ubuntu Xenial):
importance:	Undecided → Medium
Changed in keystone (Ubuntu Yakkety):
importance:	Undecided → Medium
Changed in keystone (Ubuntu Zesty):
importance:	Undecided → Medium
Changed in keystone (Ubuntu Xenial):
status:	New → Triaged
Changed in keystone (Ubuntu Yakkety):
status:	New → Triaged
Changed in keystone (Ubuntu Zesty):
status:	New → Triaged

Jorge Niedbalski (niedbalski) on 2017-05-31

Changed in cloud-archive:
assignee:	nobody → Jorge Niedbalski (niedbalski)
status:	New → In Progress
Changed in keystone (Ubuntu):
assignee:	nobody → Jorge Niedbalski (niedbalski)
importance:	Undecided → High
status:	New → In Progress
Changed in keystone (Ubuntu Xenial):
assignee:	nobody → Jorge Niedbalski (niedbalski)
status:	Triaged → In Progress
Changed in keystone (Ubuntu Yakkety):
assignee:	nobody → Jorge Niedbalski (niedbalski)
importance:	Medium → High
status:	Triaged → In Progress
Changed in keystone (Ubuntu Zesty):
assignee:	nobody → Jorge Niedbalski (niedbalski)
importance:	Medium → High
status:	Triaged → In Progress

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-05-31:

#25

fix-lp-1649616-artful.debdiff Edit (3.1 KiB, text/plain)

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-05-31:

#26

fix-lp-1649616-zesty.debdiff Edit (3.1 KiB, text/plain)

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-05-31:

#27

fix-lp-1649616-yakkety.debdiff Edit (3.1 KiB, text/plain)

Changed in keystone (Ubuntu Xenial):
importance:	Medium → High

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-05-31:

#28

fix-lp-1649616-xenial.debdiff Edit (3.1 KiB, text/plain)

Eric Desrochers (slashd) on 2017-05-31

tags:

added: sts-sru-needed

Jorge Niedbalski (niedbalski) on 2017-06-05

description:

updated

Revision history for this message

James Page (james-page) wrote on 2017-06-07:

#29

Uploaded to xenial-proposed for SRU team review.

Revision history for this message

James Page (james-page) wrote on 2017-06-07:

#30

Uploaded to yakkety and zesty proposed for SRU team review.

Eric Desrochers (slashd) on 2017-06-08

tags:

added: sts-sru-done
removed: sts-sru-needed

Revision history for this message

Edward Hope-Morley (hopem) wrote on 2017-06-08:

#31

@slashd this sru is not finished yet so i'll re-instate the -needed tag (that we use for tracking purposes)

tags:

added: sts-sru-needed
removed: sts-sru-done

Eric Desrochers (slashd) on 2017-06-08

tags:	added: sts-sru-needd removed: sts-sru-needed
tags:	added: sts-sru-needed removed: sts-sru-needd

Emilien Macchi (emilienm) on 2017-06-08

Changed in tripleo:
milestone:	pike-2 → pike-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-08: Fix included in openstack/keystone 12.0.0.0b2

#32

This issue was fixed in the openstack/keystone 12.0.0.0b2 development milestone.

Revision history for this message

Andy Whitcroft (apw) wrote on 2017-06-16: Please test proposed package

#33

Hello Alex, or anyone else affected,

Accepted keystone into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keystone/2:11.0.2-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in keystone (Ubuntu Zesty):
status:	In Progress → Fix Committed
tags:	added: verification-needed

Revision history for this message

James Page (james-page) wrote on 2017-06-19:

#34

Hello Alex, or anyone else affected,

Accepted keystone into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

sudo add-apt-repository cloud-archive:ocata-proposed
sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-ocata-needed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-23: Fix merged to keystone (stable/ocata)

#35

Reviewed: https://review.openstack.org/469069
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=6074166b29a0546c92e3a1c152370d5b39c1dafe
Submitter: Jenkins
Branch: stable/ocata

commit 6074166b29a0546c92e3a1c152370d5b39c1dafe
Author: Peter Sabaini <email address hidden>
Date: Thu Apr 6 23:06:29 2017 +0200

Make flushing tokens more robust

Commit token flushes between batches in order to lower resource
consumption and make flushing more robust for replication

    Change-Id: I9be37e420353a336a8acd820eadd47d4bcf7324f
    Closes-Bug: #1649616
    (cherry picked from commit dc7f81083180eeb5233f7007e3d2514cc0d7c6d3)

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2017-06-27: Change of SRU verification policy

#36

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

Revision history for this message

Andy Whitcroft (apw) wrote on 2017-07-03: Please test proposed package

#37

Hello Alex, or anyone else affected,

Accepted keystone into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keystone/2:10.0.1-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-yakkety to verification-done-yakkety. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-yakkety. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in keystone (Ubuntu Yakkety):
status:	In Progress → Fix Committed
tags:	added: verification-needed-yakkety
Changed in keystone (Ubuntu Xenial):
status:	In Progress → Fix Committed
tags:	added: verification-needed-xenial

Revision history for this message

Andy Whitcroft (apw) wrote on 2017-07-03:

#38

Hello Alex, or anyone else affected,

Accepted keystone into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keystone/2:9.3.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message

James Page (james-page) wrote on 2017-07-04:

#39

Hello Alex, or anyone else affected,

Accepted keystone into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

sudo add-apt-repository cloud-archive:mitaka-proposed
sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-mitaka-needed

Revision history for this message

James Page (james-page) wrote on 2017-07-04:

#40

Hello Alex, or anyone else affected,

Accepted keystone into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

sudo add-apt-repository cloud-archive:newton-proposed
sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-newton-needed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-04: Fix proposed to keystone (master)

#41

Fix proposed to branch: master
Review: https://review.openstack.org/480287

Revision history for this message

Chris Halse Rogers (raof) wrote on 2017-07-05:

#42

Is testing for this bug part of the standard test-suite run for the point-release stable update (https://bugs.launchpad.net/ubuntu/+source/swift/+bug/1696139)?

If so, please mention it so that this can be released.

If not, please test this so that the point-releases can be released :).

Emilien Macchi (emilienm) on 2017-07-05

Changed in tripleo:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-11: Fix merged to keystone (master)

#43

Reviewed: https://review.openstack.org/480287
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0b5c5c03ecb6cd261ec06b4e2465c8d88b8c1725
Submitter: Jenkins
Branch: master

commit 0b5c5c03ecb6cd261ec06b4e2465c8d88b8c1725
Author: Raildo Mascena <email address hidden>
Date: Tue Jul 4 14:10:16 2017 -0300

Fixing flushing tokens workflow

    During a backport patch [0] for this fix
    it was found some problems in the previous
    approach like, It didn't enabled back the
    session.autocommit. Another comment was we should
    create a new session and commit on it instead of
    disable/enable autocommit.

    After this, we should backport this change in order
    to fix the previous releases, instead of the other
    one.

[0] https://review.openstack.org/#/c/469514

Change-Id: Ifc024ba0e86bb71f4ab8b019917782bc5bf3be7a
Closes-Bug: #1649616

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-11: Related fix proposed to puppet-keystone (stable/newton)

#44

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/482552

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-11: Related fix proposed to tripleo-heat-templates (stable/newton)

#45

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/482585

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-11: Fix proposed to keystone (stable/ocata)

#46

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/482601

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-11: Fix proposed to keystone (stable/newton)

#47

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/482606

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-12: Related fix merged to puppet-keystone (stable/newton)

#48

Reviewed: https://review.openstack.org/482552
Committed: https://git.openstack.org/cgit/openstack/puppet-keystone/commit/?id=c1bda5f81e35ec597f8d684a35c2c1446c8b1527
Submitter: Jenkins
Branch: stable/newton

commit c1bda5f81e35ec597f8d684a35c2c1446c8b1527
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Apr 18 13:13:27 2017 +0300

Change keystone token flush to run hourly

    In a recent commit [1] the keystone token flush cron job was changed to
    run twice a day. However, this change was not enough for big
    deployments.

    After getting some customer feedback and looking at what other projects
    are doing [2] [3] [4]. It seems that running this job hourly is the way
    to go.

    [1] Ia0b0fb422318712f4b0f4d023cbb3a61d40bb85d
    [2] https://www.ibm.com/support/knowledgecenter/en/SSB27U_6.4.0/com.ibm.zvm.v640.hcpo4/exptoken.htm
    [3] https://review.openstack.org/#/c/88670/8
    [4] https://github.com/openstack/charm-keystone/blob/master/templates/keystone-token-flush

    Conflicts:
     manifests/cron/token_flush.pp
     spec/acceptance/keystone_federation_identity_provider_spec.rb
     spec/acceptance/keystone_federation_shibboleth_spec.rb
     spec/acceptance/keystone_wsgi_apache_spec.rb
     spec/classes/keystone_cron_token_flush_spec.rb

    (cherry picked from commit f694b5551f896042df6aeb751c65986ef3342f54)
    Change-Id: I6ec7ec8111bd93e5638cfe96189e36f0e0691d65
    Related-Bug: #1649616
    (cherry picked from commit 90ffc7f6008370e9d9893ab69f31683352c854c3)

tags:

added: in-stable-newton

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-07-13:

#49

This is the verification that I ran through:

1) juju run --application keystone "sudo test -f /etc/cron.d/keystone-token-flush && echo 'Exists'" --format=yaml

2) Commented /etc/cron.d/keystone-token-flush

3) Killed all the mysql threads

*) SELECT trx_mysql_thread_id as ID, trx_id, trx_query, trx_started, trx_state FROM information_schema.innodb_trx WHERE trx_state = "RUNNING" AND trx_query LIKE "DELETE FROM token%" ORDER BY trx_started;

*) KILL thread_id;
*) KILL CONNECTION thread_id;

4) Manually run the token_flush command

*) sudo -H -u keystone bash -c "/usr/bin/keystone-manage token_flush > /tmp/token-flush.log"

After running this, provide me the the following details as attachments into this case. (Please connect to the database using the before mentioned method).

5) Re-enable the cronjob, and perform the following queries on the database to check
if there were any locks or stuck transactions.

SHOW open tables WHERE In_use > 0;
SELECT * FROM information_schema.innodb_locks;
SELECT * FROM information_schema.innodb_trx ORDER BY trx_started;
SHOW ENGINE innodb status;

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-07-13:

#50

For the case of xenial/mitaka the results for 5) http://pastebin.ubuntu.com/25082901/,
which indeed shows the problem as solved

Marking the verification as done on X/M.

tags:

added: verification-done-xenial verification-mitaka-done
removed: verification-mitaka-needed verification-needed-xenial

Brian Murray (brian-murray) on 2017-07-13

tags:

added: verification-needed-zesty

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-07-13:

#51

Same results for yakkety/newton for 5) http://pastebin.ubuntu.com/25084654/

Marking the verification as done for Y/N

tags:

added: verification-done-yakkety verification-newton-done
removed: verification-needed-yakkety verification-newton-needed

Revision history for this message

Jorge Niedbalski (niedbalski) wrote on 2017-07-13:

#52

Same results for ocata (http://pastebin.ubuntu.com/25085019/)

tags:

added: verification-done verification-done-zesty verification-ocata-done
removed: verification-needed verification-needed-zesty verification-ocata-needed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-14: Fix merged to keystone (stable/ocata)

#53

Reviewed: https://review.openstack.org/482601
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=83fccfaf8dc2ae2f4b1d47b09d44cb5c995f4d52
Submitter: Jenkins
Branch: stable/ocata

commit 83fccfaf8dc2ae2f4b1d47b09d44cb5c995f4d52
Author: Raildo Mascena <email address hidden>
Date: Tue Jul 4 14:10:16 2017 -0300

Fixing flushing tokens workflow

    During a backport patch [0] for this fix
    it was found some problems in the previous
    approach like, It didn't enabled back the
    session.autocommit. Another comment was we should
    create a new session and commit on it instead of
    disable/enable autocommit.

    After this, we should backport this change in order
    to fix the previous releases, instead of the other
    one.

[0] https://review.openstack.org/#/c/469514

    Change-Id: Ifc024ba0e86bb71f4ab8b019917782bc5bf3be7a
    Closes-Bug: #1649616
    (cherry picked from commit 0b5c5c03ecb6cd261ec06b4e2465c8d88b8c1725)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-14: Fix merged to keystone (stable/newton)

#54

Reviewed: https://review.openstack.org/482606
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=058ea4262459be90f0721891b6ac2d29600e77c1
Submitter: Jenkins
Branch: stable/newton

commit 058ea4262459be90f0721891b6ac2d29600e77c1
Author: Raildo Mascena <email address hidden>
Date: Tue Jul 4 14:10:16 2017 -0300

Fixing flushing tokens workflow

    During a backport patch [0] for this fix
    it was found some problems in the previous
    approach like, It didn't enabled back the
    session.autocommit. Another comment was we should
    create a new session and commit on it instead of
    disable/enable autocommit.

    After this, we should backport this change in order
    to fix the previous releases, instead of the other
    one.

[0] https://review.openstack.org/#/c/469514

    Change-Id: Ifc024ba0e86bb71f4ab8b019917782bc5bf3be7a
    Closes-Bug: #1649616
    (cherry picked from commit 0b5c5c03ecb6cd261ec06b4e2465c8d88b8c1725)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-14: Change abandoned on keystone (stable/newton)

#55

Change abandoned by Steve Martinelli (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/469514
Reason: see referred patch in previous comment

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2017-07-17: Update Released

#56

The verification of the Stable Release Update for keystone has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-07-17:

#57

This bug was fixed in the package keystone - 2:11.0.2-0ubuntu1

---------------
keystone (2:11.0.2-0ubuntu1) zesty; urgency=medium

  [ Jorge Niedbalski ]
  * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
    between batches in order to lower resource consumption and
    make flushing more robust for replication (LP: #1649616).

[ James Page ]
* New upstream stable release for OpenStack Ocata (LP: #1696139).

-- James Page <email address hidden> Wed, 07 Jun 2017 16:01:45 +0100

Changed in keystone (Ubuntu Zesty):
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-07-17:

#58

This bug was fixed in the package keystone - 2:10.0.1-0ubuntu2

---------------
keystone (2:10.0.1-0ubuntu2) yakkety; urgency=high

  * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
    between batches in order to lower resource consumption and
    make flushing more robust for replication (LP: #1649616).

-- Jorge Niedbalski <email address hidden> Wed, 07 Jun 2017 13:07:50 +0100

Changed in keystone (Ubuntu Yakkety):
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-07-17:

#59

This bug was fixed in the package keystone - 2:9.3.0-0ubuntu2

---------------
keystone (2:9.3.0-0ubuntu2) xenial; urgency=high

  * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit
    token flushes between batches in order to lower resource
    consumption and make flushing more robust for replication
    (LP: #1649616).

-- Jorge Niedbalski <email address hidden> Wed, 07 Jun 2017 10:33:50 +0100

Changed in keystone (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Revision history for this message

James Page (james-page) wrote on 2017-07-18:

#60

The verification of the Stable Release Update for keystone has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

James Page (james-page) wrote on 2017-07-18:

#61

This bug was fixed in the package keystone - 2:11.0.2-0ubuntu1~cloud0
---------------

keystone (2:11.0.2-0ubuntu1~cloud0) xenial-ocata; urgency=medium
.
   * New upstream release for the Ubuntu Cloud Archive.
.
keystone (2:11.0.2-0ubuntu1) zesty; urgency=medium
.
   [ Jorge Niedbalski ]
   * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
     between batches in order to lower resource consumption and
     make flushing more robust for replication (LP: #1649616).
.
   [ James Page ]
   * New upstream stable release for OpenStack Ocata (LP: #1696139).

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-19: Related fix merged to tripleo-heat-templates (stable/newton)

#62

Reviewed: https://review.openstack.org/482585
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6299a169b4c17c4cc3946ebe4f66342ac2054d6b
Submitter: Jenkins
Branch: stable/newton

commit 6299a169b4c17c4cc3946ebe4f66342ac2054d6b
Author: Harry Rybacki <email address hidden>
Date: Wed Jul 12 13:25:30 2017 +0000

Run token flush cron job hourly by default

    Running this job once a day has proven problematic for large
    deployments as seen in the bug report. Setting it to run hourly
    would be an improvement to the current situation, as the flushes
    wouldn't need to process as much data.

Note that this only affects people using UUID as the token provider.

    Change-Id: I462e4da2bfdbcba0403ecde5d613386938e2283a
    Related-Bug: #1649616
    (cherry picked from commit 65e643aca2202f031db94f1ccd3d44e195e5e772)

Revision history for this message

James Page (james-page) wrote on 2017-07-25: Update Released

#63

The verification of the Stable Release Update for keystone has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

James Page (james-page) wrote on 2017-07-25:

#64

This bug was fixed in the package keystone - 2:9.3.0-0ubuntu2~cloud0
---------------

keystone (2:9.3.0-0ubuntu2~cloud0) trusty-mitaka; urgency=medium
.
   * New update for the Ubuntu Cloud Archive.
.
keystone (2:9.3.0-0ubuntu2) xenial; urgency=high
.
   * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit
     token flushes between batches in order to lower resource
     consumption and make flushing more robust for replication
     (LP: #1649616).

Revision history for this message

James Page (james-page) wrote on 2017-07-25:

#65

The verification of the Stable Release Update for keystone has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

James Page (james-page) wrote on 2017-07-25:

#66

This bug was fixed in the package keystone - 2:10.0.1-0ubuntu2~cloud0
---------------

keystone (2:10.0.1-0ubuntu2~cloud0) xenial-newton; urgency=medium
.
   * New update for the Ubuntu Cloud Archive.
.
keystone (2:10.0.1-0ubuntu2) yakkety; urgency=high
.
   * d/p/0001-Make-flushing-tokens-more-robust.patch: Commit token flushes
     between batches in order to lower resource consumption and
     make flushing more robust for replication (LP: #1649616).

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-26: Fix included in openstack/keystone 11.0.3

#67

This issue was fixed in the openstack/keystone 11.0.3 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-26: Fix included in openstack/keystone 10.0.3

#69

This issue was fixed in the openstack/keystone 10.0.3 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-28: Fix included in openstack/keystone 12.0.0.0b3

#70

This issue was fixed in the openstack/keystone 12.0.0.0b3 development milestone.

Emilien Macchi (emilienm) on 2017-07-30

Changed in tripleo:
milestone:	pike-3 → pike-rc1

Emilien Macchi (emilienm) on 2017-07-31

Changed in tripleo:
milestone:	pike-rc1 → queens-1

Juan Antonio Osorio Robles (juan-osorio-robles) on 2017-07-31

Changed in tripleo:
status:	In Progress → Fix Released

Edward Hope-Morley (hopem) on 2017-08-07

tags:

added: sts-sru-done
removed: sts-sru-needed

Corey Bryant (corey.bryant) on 2017-08-31

Changed in keystone (Ubuntu):
status:	In Progress → Invalid
Changed in cloud-archive:
status:	In Progress → Invalid

Tobias Urdin (tobias-urdin) on 2018-04-04

Changed in puppet-keystone:
status:	Triaged → Fix Released

Revision history for this message

Morgan Fainberg (mdrnstm) wrote on 2018-10-24:

#71

newton is EOL

OpenStack Identity (keystone)

Keystone Token Flush job does not complete in HA deployed environment

Bug Description

Related branches

Duplicates of this bug

Other bug subscribers

Patches

Bug attachments

Remote bug watches

	Status	Importance	Assigned to	Milestone
OpenStack Identity (keystone)	Fix Released	Medium	Peter Sabaini
Newton	Won't Fix	Medium	Raildo Mascena de Sousa Filho
Ocata	Fix Released	Medium	Raildo Mascena de Sousa Filho
Ubuntu Cloud Archive	Invalid	Undecided	Jorge Niedbalski
Mitaka	Fix Released	Medium	Jorge Niedbalski
Newton	Fix Released	Medium	Jorge Niedbalski
Ocata	Fix Released	Medium	Jorge Niedbalski
puppet-keystone	Fix Released	Medium	Juan Antonio Osorio Robles
tripleo	Fix Released	Medium	Juan Antonio Osorio Robles	tripleo queens-1
keystone (Ubuntu)	Invalid	High	Jorge Niedbalski
Xenial	Fix Released	High	Jorge Niedbalski
Yakkety	Fix Released	High	Jorge Niedbalski
Zesty	Fix Released	High	Jorge Niedbalski