Redis caching backend throws stacktraces

[Christoph Manns]

If you use redis as a caching backend and you delete a server with
no consoleauth tokens you'll get a stacktrace as an empty list is
passed down to the redis client and ultimately the redis server
which responds with an error, complaining about a wrong number
of arguments for the del command.

2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server [req-cc86f95f-00aa-428a-8920-4f0ea478887b a6f683db07b747188336d9a2ebe1c1c2 0e8a0eaa143c4af3938db81fc9df5abf - default default] Exception during message handling: ResponseError: wrong number of arguments for 'del' command
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/consoleauth/manager.py", line 142, in delete_tokens_for_instance
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server [tok.encode('UTF-8') for tok in tokens])
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/cache_utils.py", line 130, in delete_multi
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server return self.region.delete_multi(keys)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1017, in delete_multi
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server self.backend.delete_multi(keys)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/redis.py", line 183, in delete_multi
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server self.client.delete(*keys)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/redis/client.py", line 824, in delete
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server return self.execute_command('DEL', *names)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server return self.parse_response(connection, command_name, **options)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/redis/client.py", line 577, in parse_response
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server response = connection.read_response()
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/redis/connection.py", line 574, in read_response
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server raise response
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server ResponseError: wrong number of arguments for 'del' command
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server

[Matt Riedemann (mriedem)]

https://review.openstack.org/#/c/605748/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605748
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e5884a8fd51b735cf870f818d2a828163006e18a
Submitter: Zuul
Branch: master

commit e5884a8fd51b735cf870f818d2a828163006e18a
Author: Christoph Manns <email address hidden>
Date: Thu Sep 27 16:18:54 2018 +0200

    Fix stacktraces with redis caching backend

    If you use redis as a caching backend and you delete a server with
    no consoleauth tokens you'll get a stacktrace as an empty list is
    passed down to the redis client and ultimately the redis server
    which responds with an error, complaining about a wrong number
    of arguments for the del command.
    The code now checks if the list of tokens is empty and only calls
    the caching backend if there are tokens available to delete.
    This also may improve performance, as it no longer hands down an
    empty list.

    Closes-Bug: #1794812
    Change-Id: Iffdd4e251bfa2bac1bfd49498e32b738843709de

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/606895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/606896

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/606895
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=745f0b6da4f65146073f16c8b1e6809f348d2652
Submitter: Zuul
Branch: stable/rocky

commit 745f0b6da4f65146073f16c8b1e6809f348d2652
Author: Christoph Manns <email address hidden>
Date: Thu Sep 27 16:18:54 2018 +0200

    Fix stacktraces with redis caching backend

    If you use redis as a caching backend and you delete a server with
    no consoleauth tokens you'll get a stacktrace as an empty list is
    passed down to the redis client and ultimately the redis server
    which responds with an error, complaining about a wrong number
    of arguments for the del command.
    The code now checks if the list of tokens is empty and only calls
    the caching backend if there are tokens available to delete.
    This also may improve performance, as it no longer hands down an
    empty list.

    Closes-Bug: #1794812
    Change-Id: Iffdd4e251bfa2bac1bfd49498e32b738843709de
    (cherry picked from commit e5884a8fd51b735cf870f818d2a828163006e18a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/606896
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c46bd362e5388eadcc294ef24b7f9bb1204f641
Submitter: Zuul
Branch: stable/queens

commit 7c46bd362e5388eadcc294ef24b7f9bb1204f641
Author: Christoph Manns <email address hidden>
Date: Thu Sep 27 16:18:54 2018 +0200

    Fix stacktraces with redis caching backend

    If you use redis as a caching backend and you delete a server with
    no consoleauth tokens you'll get a stacktrace as an empty list is
    passed down to the redis client and ultimately the redis server
    which responds with an error, complaining about a wrong number
    of arguments for the del command.
    The code now checks if the list of tokens is empty and only calls
    the caching backend if there are tokens available to delete.
    This also may improve performance, as it no longer hands down an
    empty list.

    Closes-Bug: #1794812
    Change-Id: Iffdd4e251bfa2bac1bfd49498e32b738843709de
    (cherry picked from commit e5884a8fd51b735cf870f818d2a828163006e18a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.9 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.