radosgw doesn't start a new log on SIGHUP

[Sergii Golovatiuk]

Hi,

radosgw service doesn't produce a new log after logrotation. After investigation I found that SIGHUP doesn't work proplerly. The issue is present on CentOS (Ubuntu is not affected). Here is a way how to reproduce

root@node-1:~# lsof -p `pgrep radosgw` | grep ceph-client.radosgw.gateway.log
radosgw 19420 root 3w REG 252,0 45554 146958 /var/log/radosgw/ceph-client.radosgw.gateway.log

As you can see file is present. Ok. let's delete it. that's what usually logrotate does

root@node-1:~# lsof -p `pgrep radosgw` | grep ceph-client.radosgw.gateway.log
radosgw 19420 root 3w REG 252,0 120611 146958 /var/log/radosgw/ceph-client.radosgw.gateway.log (deleted)

file has been marked as deleted. Ok. let's send a SIGHUP
root@node-1:~# lsof -p `pgrep radosgw` | grep ceph-client.radosgw.gateway.log

as we can see radosgw didn't start writing logs to a new file

[Aleksander Mogylchenko (amogylchenko)]

radosgw works from root user in Ubuntu, while it uses apache under CentOS - which has no rights to write to /var/log/radosgw folder.

[Aleksander Mogylchenko (amogylchenko)]

You can find logs in /var/log/syslog folder.

[Aleksander Mogylchenko (amogylchenko)]

btw, not sure why it is High and assigned to 6.1 - location of logs is a minor issue.

[Aleksander Mogylchenko (amogylchenko)]

/var/log/syslog *file*, not a folder - that was a typo.

[Dmitry Borodaenko (angdraug)]

It's High because failure to rotate logs means eventually /var/log will run out of disk space. But I agree that it's not Critical and so shouldn't be targeted for 6.1, 6.1-updates at best.

[Kostiantyn Danylov (kdanylov)]

Dmitry, radosgw would stop writting logs at all, as I understand from investigation. So we would lost a logs, not fill up drive

[Aleksander Mogylchenko (amogylchenko)]

If one sets user to root:
[root@node-4 radosgw]# grep user /etc/ceph/ceph.conf
user = root

Everything works fine:
root@node-4 radosgw]# lsof -p 32179 | grep log
radosgw 32179 root cwd DIR 253,1 4096 385044 /var/log/radosgw
radosgw 32179 root 3w REG 253,1 5680 384892 /var/log/radosgw/ceph-client.radosgw.gateway.log
[root@node-4 radosgw]# lsof -p 32179 | grep radosgw.gateway.log
radosgw 32179 root 3w REG 253,1 6665 384892 /var/log/radosgw/ceph-client.radosgw.gateway.log
[root@node-4 radosgw]# rm -rf /var/log/radosgw/ceph-client.radosgw.gateway.log
[root@node-4 radosgw]# kill -HUP 32179
[root@node-4 radosgw]# lsof -p 32179 | grep radosgw.gateway.log
radosgw 32179 root 3w REG 253,1 394 384892 /var/log/radosgw/ceph-client.radosgw.gateway.log

[Aleksander Mogylchenko (amogylchenko)]

This issue is different from https://bugs.launchpad.net/fuel/+bug/1465708, please do not mix them.

[Dmitry Borodaenko (angdraug)]

Good point. Not writing any logs is almost as bad as filling up the logs partition so that no services can write any logs, but still not a release blocker.

[Baboune (seyvet)]

I confirm that writing no logs is the behavior we have observed in our environment as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/205045

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Pawel Stefanski <email address hidden>
Review: https://review.fuel-infra.org/9832

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pawel Stefanski <email address hidden> on branch: master
Review: https://review.fuel-infra.org/9832
Reason: wrong branch

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Pawel Stefanski <email address hidden>
Review: https://review.fuel-infra.org/9843

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pawel Stefanski <email address hidden> on branch: master
Review: https://review.fuel-infra.org/9843
Reason: wrong branch

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Pawel Stefanski <email address hidden>
Review: https://review.fuel-infra.org/9844

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 7.0
Change author: Pawel Stefanski <email address hidden>
Review: https://review.fuel-infra.org/9845

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pawel Stefanski <email address hidden> on branch: master
Review: https://review.fuel-infra.org/9844
Reason: wrong branch

[OpenStack Infra (hudson-openstack)]

Change abandoned by Pawel Stefanski (<email address hidden>) on branch: master
Review: https://review.openstack.org/205045
Reason: Addressed in package patch.

[Fuel Devops McRobotson (fuel-devops-robot)]

Change restored by Mateusz Matuszkowiak <email address hidden> on branch: master
Review: https://review.fuel-infra.org/9844

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Mateusz Matuszkowiak <email address hidden> on branch: 7.0
Review: https://review.fuel-infra.org/9845

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Mateusz Matuszkowiak <email address hidden> on branch: master
Review: https://review.fuel-infra.org/9844

[Aleksandr Didenko (adidenko)]

All the patches related to this bug are abandoned. So I'm changing status to Confirmed

[Bogdan Dobrelya (bogdando)]

We should not workaround SIGHUP processing issues with puppet, moved to MOS Ceph team

[Alyona Kiseleva (akiselyova)]

Aleksander Mogylchenko have already posted the right reason: apache user, who run radosgw in CentOS, have no rights to create file in /var/log folder.