logrotate on master node fails

[Sergii Golovatiuk]

I have found that all docker containers have

crond, xinetd, rsyslogd inside of containers.

This creates problems with logrotate we do on masternode with

# bash -x /etc/logrotate.d/fuel.nodaily

For instance logrotate for nginx inside of container have problems with rotating nginx logs. Just run

# pgrep nginx
294

# cd /proc/294/fd
# ll
total 0
lrwx------ 1 root root 64 Jan 4 15:20 0 -> /dev/null
lrwx------ 1 root root 64 Jan 4 15:20 1 -> /dev/null
lrwx------ 1 root root 64 Jan 4 15:20 10 -> socket:[100990]
lrwx------ 1 root root 64 Jan 4 15:20 11 -> socket:[100991]
lrwx------ 1 root root 64 Jan 4 15:20 12 -> socket:[100992]
lrwx------ 1 root root 64 Jan 4 15:20 13 -> socket:[103809]
lrwx------ 1 root root 64 Jan 4 15:20 14 -> socket:[103810]
lrwx------ 1 root root 64 Jan 4 15:20 15 -> socket:[103811]
lrwx------ 1 root root 64 Jan 4 15:20 16 -> socket:[103812]
lrwx------ 1 root root 64 Jan 4 15:20 17 -> socket:[103813]
lrwx------ 1 root root 64 Jan 4 15:20 18 -> socket:[103814]
lrwx------ 1 root root 64 Jan 4 15:20 19 -> socket:[103815]
l-wx------ 1 root root 64 Jan 4 15:20 2 -> /var/log/nginx/error.log-20151230 (deleted)
lrwx------ 1 root root 64 Jan 4 15:20 3 -> socket:[103808]
l-wx------ 1 root root 64 Jan 4 15:20 4 -> /var/log/nginx/error.log-20151230 (deleted)
l-wx------ 1 root root 64 Jan 4 15:20 5 -> /var/log/nginx/access.log
l-wx------ 1 root root 64 Jan 4 15:20 6 -> /var/log/nginx/access_nailgun.log-20151230 (deleted)
l-wx------ 1 root root 64 Jan 4 15:20 7 -> /var/log/nginx/error_nailgun.log-20151229
l-wx------ 1 root root 64 Jan 4 15:20 8 -> /var/log/nginx/access_repo.log-20151230 (deleted)
l-wx------ 1 root root 64 Jan 4 15:20 9 -> /var/log/nginx/error_repo.log-20151230 (deleted)

# cd /var/log/nginx/
# ll
total 4056160
-rw-r--r-- 1 root root 0 Dec 29 13:14 access.log
-rw-r--r-- 1 nginx nginx 0 Dec 30 03:06 access_nailgun.log
-rw-r--r-- 1 root root 5735128 Dec 30 03:05 access_nailgun.log-20151230
-rw-r--r-- 1 nginx nginx 0 Dec 30 03:06 access_repo.log
-rw-r--r-- 1 root root 43410 Dec 29 20:10 access_repo.log-20151230.gz
-rw-r--r-- 1 nginx nginx 0 Dec 30 03:06 error.log
-rw-r--r-- 1 root root 221 Dec 29 13:14 error.log-20151230.gz
-rw-r--r-- 1 nginx nginx 0 Dec 29 14:06 error_nailgun.log
-rw-r--r-- 1 root root 4145066270 Jan 4 16:22 error_nailgun.log-20151229
-rw-r--r-- 1 nginx nginx 0 Dec 30 03:06 error_repo.log
-rw-r--r-- 1 root root 135075 Dec 30 03:04 error_repo.log-20151230.gz
-rw-r--r-- 1 root root 955733 Dec 29 19:59 error_repo.log.1.gz
-rw-r--r-- 1 root root 790415 Dec 29 18:59 error_repo.log.2.gz
-rw-r--r-- 1 root root 758530 Dec 29 16:59 error_repo.log.3.gz

It looks like copy truncate doesn't work properly.

[Sergii Golovatiuk (sgolovatiuk)]

It looks like logrotate inside of containers interacts with bash -x /etc/logrotate.d/fuel.nodaily

Firsly, we should disable logrotate in containers.

[Sergii Golovatiuk (sgolovatiuk)]

If we look at containers precisely they run a lot of services that must be disabled by default. It kills a container way. xinetd, crond, rsyslogd must be stopped and removed from container. They were started by multi-user.target

I see the most easiest way is to change Dockerfile as

RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ “/sys/fs/cgroup” ]
CMD [“/usr/sbin/init”]

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/266302

[Matthew Mosesohn (raytrac3r)]

These packages are installed in the base container, but shouldn't be enabled by default. We have a very very narrow scope of what we want to start automatically. It looks like we need to default to disabling these when we build the base container.

[Matthew Mosesohn (raytrac3r)]

#3 addresses my comments in #4 :) Now to gather reviews

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/266302
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=2d9d2953c143509d49bdf4c7271d34c56e77a6f3
Submitter: Jenkins
Branch: master

commit 2d9d2953c143509d49bdf4c7271d34c56e77a6f3
Author: Ivan Suzdal <email address hidden>
Date: Tue Jan 12 14:28:55 2016 +0300

    Disable crond,rsyslogd,xinetd by default in containers.

    Change-Id: Ifde2dd441dae75068f47a2f2bc38479c0b07116e
    Related-Bug: #1530915

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/267655

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267655
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=a003837c4d1bb2891ae04a58b1c0d9b6337b2f3a
Submitter: Jenkins
Branch: stable/8.0

commit a003837c4d1bb2891ae04a58b1c0d9b6337b2f3a
Author: Ivan Suzdal <email address hidden>
Date: Tue Jan 12 14:28:55 2016 +0300

    Disable crond,rsyslogd,xinetd by default in containers.

    Change-Id: Ifde2dd441dae75068f47a2f2bc38479c0b07116e
    Related-Bug: #1530915
    (cherry picked from commit 2d9d2953c143509d49bdf4c7271d34c56e77a6f3)

[Ksenia Svechnikova (kdemina)]

Verify the issue on ISO#461