Memcached failed to start with "/var/log/memcached.log: Permission denied" reboot.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Bogdan Dobrelya |
Bug Description
Hi,
originally found there https:/
On a 3 nodes setup, after a update, restart, scale out and re-deploy ,
some[1] of the memcached container fail to restart with that error:
[root@
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
/bin/bash: /var/log/
The container is in that status:
fa8fc2bf967d 192.168.
each line correspond to a restart.
On the other two controllers the restart was successful.
So hold your belt tight as it's going to be hairy....
The log directory /var/log/
/var/log where we expect memcached to write its log by adding
">>/var/
the docker_config step1 config we init the log like this[2]:
command: ['/bin/bash', '-c', 'source /etc/sysconfig/
so far so good ...
But if you look at any memcached directory where the container were
successfully started, then you have[3]:
- /var/log/
- /var/log/
with log.1 being opened by the memcached process:
()[
lsof: WARNING: can't stat() xfs file system /etc/sysconfig/
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
memcached 7 memcached 1w REG 252,2 41994775 97206843 /var/log/
memcached 7 memcached 2w REG 252,2 41994775 97206843 /var/log/
There is this error about /etc/sysconfig/
come back to it later. For now I just don't understand how those
files are created. "log.1" and "log.2" ... it's even more puzzling
as I can't create any file in the /var/log/ directory with permission
denied on a working memcached container.
[root@
()[
-rw-r--r--. 1 memcached root 15M Jun 13 19:31 /var/log/
-rw-r--r--. 1 memcached root 42M Jun 14 10:05 /var/log/
()[
drwxr-xr-x. 2 root root 52 Jun 13 22:01 /var/log/
()[
touch: cannot touch '/var/log/
()[
uid=
and the "log.1" does belong to memcached process:
()[
lsof: WARNING: can't stat() xfs file system /etc/sysconfig/
Output information may be incomplete.
()[
lsof: WARNING: can't stat() xfs file system /etc/sysconfig/
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
memcached 7 memcached 1w REG 252,2 43203747 97206843 /var/log/
memcached 7 memcached 2w REG 252,2 43203747 97206843 /var/log/
On the host:
[root@
total 68M
-rw-r--r--. 1 42457 root 15M Jun 13 19:31 memcached.log.2
-rw-r--r--. 1 42457 root 42M Jun 14 10:19 memcached.log.1
[root@
drwxr-xr-x. 2 root root 52 Jun 13 22:01 /var/log/
Which makes no sense.
So there are two unknown here:
1. who creates log.1 and rotate log.2 ?
2. how come the process can write in a directory that it doesn't have
write access to.
I'm at loss for question 1. For question 2 I guess that the mounting
sequence is flawed somehow, as the /etc/sysconfig/
the lsof output seems to indicate. Here more about it, the file
system /etc/sysconfig/
# docker exec -ti memcached bash
()[
lsof: WARNING: can't stat() xfs file system /etc/sysconfig/
Output information may be incomplete.
()[
PORT="11211"
USER=
MAXCONN="8192"
CACHESIZE=
OPTIONS="-vv -l 172.17.1.15 -U 0 -t 8 >> /var/log/
()[
This tends to confirm that something is wrong in the mounting
sequence.
All in all I think we should change the way the container are created to make them
more robust after any reboot operation.
If somebody has hint at what is happening there that would be awesome ... and thanks
to read till the end.
[1] first iteration of the test found 2 memcached failing to restart,
on another run only 1 fail to restart and on another (whithout the
scale out and the reconfig) all three restarted successfully. I can
find the "log.1" in a deployment as well.
[2] https:/
[3] This seems to happen only on rhel platform as in the CI jobs memcached.log seems to be created fine.
Changed in tripleo: | |
importance: | High → Critical |
Changed in tripleo: | |
assignee: | Bogdan Dobrelya (bogdando) → Sofer Athlan-Guyot (sofer-athlan-guyot) |
Changed in tripleo: | |
assignee: | Sofer Athlan-Guyot (sofer-athlan-guyot) → Emilien Macchi (emilienm) |
Changed in tripleo: | |
assignee: | Emilien Macchi (emilienm) → Bogdan Dobrelya (bogdando) |
no longer affects: | tripleo/queens |
Related fix proposed to branch: master /review. openstack. org/575409
Review: https:/