"Unable to add token to revocation list" warning happened when revoking token in memcache

[Eric Zhou]

Memcache backend is used to store the token. When revoking a token, such error reported.
"Unable to add token to revocation list"

As a result, the revoked token could not be added to revocation-list in memcache although the token was actually revoked.
I found this warning always happen when the size of value of the revocation-list key in memcache is about 512K.

Expected result:
No warning exception should be raised when revoking token.

[mouadino (mouadino)]

Just stumble upon the same problem while running tempest on an openstack installation that use memcached as keystone backend, and there is my analyze of the situation.

As far as i can tell, the problem is the way the revoked token are stored in memcached, basically each revoked token is appended to this item "revocation-list" and because memcached has a default max_item_size set to 1MB (echo 'stats settings' | nc localhost 11211 | grep 'item_size_max') than as soon as this limit is hit keystone will start raising an error when it try to append to this item.

Code taken from keystone/token/backends/memcache.py:

def _add_to_revocation_list(self, data):
        data_json = jsonutils.dumps(data)
        if not self.client.append(self.revocation_key, ',%s' % data_json):
            if not self.client.add(self.revocation_key, data_json):
                if not self.client.append(self.revocation_key,
                                          ',%s' % data_json):
                    msg = _('Unable to add token to revocation list.')
                    raise exception.UnexpectedError(msg)

The quick and dirty and temporary fix (which is of course not recommended) is to change the memcached default max_item_size to something bigger than 1MB and thus is possible only with memcached 1.4.2 and above by supplying the -I (capital i) argument.

   $ memcached -I 10m ... # max_item_size = 10MB

This will increase heavily memory consumption of memcached, that's one of the reason why it's not recommended.

A permanent fix will be when this https://blueprints.launchpad.net/keystone/+spec/revocation-backend will be implemented i guess.

A lesson to take from this is that memcached is not meant to store big lists.

[Morgan Fainberg (mdrnstm)]

This will be addressed by the new revocation event system: https://blueprints.launchpad.net/keystone/+spec/revocation-events

[Victor Morales (electrocucaracha)]

Just by curiosity, in the function displayed above, the first and third "if" sentences are identical, is that correct?

def _add_to_revocation_list(self, data):
        data_json = jsonutils.dumps(data)
        if not self.client.append(self.revocation_key, ',%s' % data_json):
            if not self.client.add(self.revocation_key, data_json):
                if not self.client.append(self.revocation_key,
                                          ',%s' % data_json):
                    msg = _('Unable to add token to revocation list.')
                    raise exception.UnexpectedError(msg)

[Dolph Mathews (dolph)]

@Victor: good question, I was wondering the same. At minimum, the combination requires an inline comment to justify itself.

[mouadino (mouadino)]

@Victor, @Dolph: The pattern used there is from here: https://code.google.com/p/memcached/wiki/NewProgrammingTricks#Managing_lists_with_append_/_prepend

[Adam Young (ayoung)]

Moving to Fernet tokens. Revocations will be handled by revocation events, not revocation list. Memcache as a storage mechanism for PKI tokens was deeply flawed, as dropping tokens from Memcache effectively unrevoked them.