Redis fails to start on controllers

Bug #1666171 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned

Bug Description

Redis fails to start on controllers in all TripleO jobs. This breaks ceilometer and gnocchi, making them to overload logs with reconnection tries and make logs huge size.

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/be399f3/logs/overcloud-controller-0/var/log/redis/redis.txt.gz
34211:M 18 Feb 07:38:39.243 # Opening Unix socket: bind: No such file or directory

Feb 18 07:18:37 localhost systemd: Started Redis persistent key-value database.
Feb 18 07:18:37 localhost systemd: Starting Redis persistent key-value database...
Feb 18 07:18:37 localhost systemd: redis.service: main process exited, code=exited, status=1/FAILURE
Feb 18 07:18:37 localhost redis-shutdown: Could not connect to Redis at 192.168.24.16:6379: Connection refused
Feb 18 07:18:37 localhost systemd: redis.service: control process exited, code=exited status=1
Feb 18 07:18:37 localhost systemd: Unit redis.service entered failed state.
Feb 18 07:18:37 localhost systemd: redis.service failed.

/etc/redis.conf:

daemonize yes
pidfile /var/run/redis/redis-server.pid
port 6379
tcp-backlog 511
bind 192.168.24.16
unixsocket /var/run/redis/redis.sock
unixsocketperm 755
timeout 0
tcp-keepalive 0
loglevel notice
logfile /var/log/redis/redis.log
syslog-enabled no
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis/
masterauth zBnyqkYQAXRyBbuQhhV6Mbs3H
slave-serve-stale-data yes
slave-read-only yes
repl-timeout 60
repl-disable-tcp-nodelay no
repl-backlog-size 1mb
repl-backlog-ttl 3600
slave-priority 100
min-slaves-to-write 0
min-slaves-max-lag 10
requirepass zBnyqkYQAXRyBbuQhhV6Mbs3H
maxclients 10000
appendonly no
appendfilename appendonly.aof
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

Changed in tripleo:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Javier Peña (jpena-c) wrote :

This has been caused by a recent redis update in the dependency repo to 3.2.7-1 [1], which misses the fixes from version 3.2.4-2 [2], namely:

- Install tmpfiles and /run/redis for legacy configurations

We need an updated redis package with the fix.

[1] http://cbs.centos.org/koji/buildinfo?buildID=15444
[2] http://cbs.centos.org/koji/buildinfo?buildID=13831

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

Was updated to redis-3.2.8-1.el7.x86_64 and started to work.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.