Comment 1 for bug 1242620

Revision history for this message
mouadino (mouadino) wrote :

Just stumble upon the same problem while running tempest on an openstack installation that use memcached as keystone backend, and there is my analyze of the situation.

As far as i can tell, the problem is the way the revoked token are stored in memcached, basically each revoked token is appended to this item "revocation-list" and because memcached has a default max_item_size set to 1MB (echo 'stats settings' | nc localhost 11211 | grep 'item_size_max') than as soon as this limit is hit keystone will start raising an error when it try to append to this item.

Code taken from keystone/token/backends/memcache.py:

def _add_to_revocation_list(self, data):
        data_json = jsonutils.dumps(data)
        if not self.client.append(self.revocation_key, ',%s' % data_json):
            if not self.client.add(self.revocation_key, data_json):
                if not self.client.append(self.revocation_key,
                                          ',%s' % data_json):
                    msg = _('Unable to add token to revocation list.')
                    raise exception.UnexpectedError(msg)

The quick and dirty and temporary fix (which is of course not recommended) is to change the memcached default max_item_size to something bigger than 1MB and thus is possible only with memcached 1.4.2 and above by supplying the -I (capital i) argument.

   $ memcached -I 10m ... # max_item_size = 10MB

This will increase heavily memory consumption of memcached, that's one of the reason why it's not recommended.

A permanent fix will be when this https://blueprints.launchpad.net/keystone/+spec/revocation-backend will be implemented i guess.

A lesson to take from this is that memcached is not meant to store big lists.