non-pacemaker redis container does not survive a reboot

[Alex Schultz]

The redis container when deployed without pacemaker does not restart on a reboot. It enters a restart loop because it cannot access /var/run/redis/redis.sock. It appears this folder is not properly mounted to the container.

[root@undercloud config-data]# cat /var/log/containers/redis/redis.log
                _._
           _.-``__ ''-._
      _.-`` `. `_. ''-._ Redis 3.2.8 (00000000/0) 64 bit
  .-`` .-```. ```\/ _.,_ ''-._
 ( ' , .-` | `, ) Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
 | `-._ `._ / _.-' | PID: 6
  `-._ `-._ `-./ _.-' _.-'
 |`-._`-._ `-.__.-' _.-'_.-'|
 | `-._`-._ _.-'_.-' | http://redis.io
  `-._ `-._`-.__.-'_.-' _.-'
 |`-._`-._ `-.__.-' _.-'_.-'|
 | `-._`-._ _.-'_.-' |
  `-._ `-._`-.__.-'_.-' _.-'
      `-._ `-.__.-' _.-'
          `-._ _.-'
              `-.__.-'

6:M 16 May 20:30:19.786 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
6:M 16 May 20:30:19.786 # Server started, Redis version 3.2.8
6:M 16 May 20:30:19.786 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
6:M 16 May 20:30:19.787 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
6:M 16 May 20:30:19.787 * The server is now ready to accept connections on port 6379
6:M 16 May 20:30:19.787 * The server is now ready to accept connections at /var/run/redis/redis.sock
<reboot>
6:M 17 May 15:20:49.704 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:20:50.390 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:20:51.044 # Opening Unix socket: bind: No such file or directory
7:M 17 May 15:20:51.766 # Opening Unix socket: bind: No such file or directory
7:M 17 May 15:20:52.846 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:19:28.846 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:19:32.298 # Opening Unix socket: bind: No such file or directory
7:M 17 May 15:19:38.955 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:19:51.997 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:20:17.853 # Opening Unix socket: bind: No such file or directory
7:M 17 May 15:21:09.308 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:22:51.970 # Opening Unix socket: bind: No such file or directory
7:M 17 May 15:26:17.022 # Opening Unix socket: bind: No such file or directory
7:M 17 May 15:33:06.877 # Opening Unix socket: bind: No such file or directory
6:M 17 May 15:46:46.333 # Opening Unix socket: bind: No such file or directory
7:M 17 May 16:14:05.000 # Opening Unix socket: bind: No such file or directory
7:M 17 May 17:08:42.064 # Opening Unix socket: bind: No such file or directory
7:M 17 May 18:57:55.934 # Opening Unix socket: bind: No such file or directory

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/569256

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/569256
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1d9629ec0b3320bcbc5a4150c8be19c6eb4096eb
Submitter: Zuul
Branch: master

commit 1d9629ec0b3320bcbc5a4150c8be19c6eb4096eb
Author: Alex Schultz <email address hidden>
Date: Thu May 17 14:54:11 2018 -0600

    Mount /var/run/redis for the redis socket

    Under pacemaker, it manages this mount correctly. When not deploying
    redis with pacemaker, the /var/run/redis folder is not properly mounted
    so the container cannot start correctly after a reboot.

    Change-Id: I1b86bc91e7ca95a1e70fecee1b6507787d98e9cd
    Closes-Bug: #1771888

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/571758

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571758
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=27265f9f856773d4def6f4a9158e3754871e04f8
Submitter: Zuul
Branch: stable/queens

commit 27265f9f856773d4def6f4a9158e3754871e04f8
Author: Alex Schultz <email address hidden>
Date: Thu May 17 14:54:11 2018 -0600

    Mount /var/run/redis for the redis socket

    Under pacemaker, it manages this mount correctly. When not deploying
    redis with pacemaker, the /var/run/redis folder is not properly mounted
    so the container cannot start correctly after a reboot.

    Change-Id: I1b86bc91e7ca95a1e70fecee1b6507787d98e9cd
    Closes-Bug: #1771888
    (cherry picked from commit 1d9629ec0b3320bcbc5a4150c8be19c6eb4096eb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b3 development milestone.