Horizon require memcached to flush data on upgrade

[Gaël THEROND]

From time to time when performing an upgrade you can end up with a not working horizon instance due to stale data on memcached.

This result in a constantly rebooting horizon container with the following error:

https://paste.opendev.org/show/bMcQB4BZh7y0FDnqsLhu/

As tracked down in here: https://access.redhat.com/solutions/7011005

The issue is memcached stale data that need to be flushed.
In our context, we can't just restart memcached instance as they kept cache alive even across docker restart.

In order to solve this issue with as less as downtime possible we have two main solutions:

1°/ - We serially flush all deployed memcached instance using the memcached flush_all command feature, however it require netcat or equivalent (nc/socat/etc) to inject the command as follow: echo 'flush_all' > nc <host_api_net> 11211

2°/ - We drop and re-instanciate the memcached container serially during the horizon upgrade task.

What do you prefer?

[Maksim Malchuk (mmalchuk)]

just to clarify one of the solution (2): drop and re-instanciate means not simple stop/start docker containers?