Take care of open and unlinked files (optimize descriptors and disk space usage)

[Bogdan Dobrelya]

We need to introduce periodic restarts for openstack services, mysql, corosync, and may be some other services as well.
(too bad Openstack services and Corosync do not understand 'kill -HUP' and became dead instead of HUPed :-) )

Please check http://pastebin.com/gUnjC0WD output for more details.

Solution:
Introduce SIGHUP support for all OSt services which missing it (https://bugs.launchpad.net/oslo/+bug/1276694)

W/a (kludge) suggested:
make '/etc/logrotate.d/10-fuel.conf' (daily logrotate job) to ensure periodic restarts for services mentioned above in order to clean their open file descriptors and to optimize disk space usage as well.
Note:
1) restarting will bring service accessibility outages for a very short amount of time.
2) Corosync gracefull restarting might be a *big* problem - it might hang for a long time (up to a half of a hour, or even hang for ever), but if it had killed -9, would cause the stonish-ng to consider such node as a failed one, thus immediately fenced (for now we do not have fencing yet, but we will in the nearest future)...

Note: to estimate the size (in bytes) of the disk space was held by deleted files at the 'node-foo' you can use:
ssh node-foo "lsof +L1 -s | awk '/\d+/ {print $7}' | awk '{sum+=$1} END {print sum}'"

[Dmitry Pyzhov (dpyzhov)]

Should it be exposed as openstack bugs?

[Bogdan Dobrelya (bogdando)]

Only the absence of SIGHUP support

[Bogdan Dobrelya (bogdando)]

related issue for reload/SIGHUP support in Oslo-incubator https://bugs.launchpad.net/oslo/+bug/1276694

[Bogdan Dobrelya (bogdando)]

The following Mirantis Openstack packages require to sync SIGHUP support from upstream:
python-nova, python-cinder, python-neutron

The following upstream and Mirantis Openstack packages require to sync SIGHUP support from Oslo (https://github.com/openstack/oslo-incubator/commit/825ace5581fbb416944acae62f51c489ed93b9c9)
python-glance, murano-conductor, murano-api

That would allow us to issue 'service foo reload' / 'kill -HUP $(pgrep foo)' for correct file descriptors handling on rotations etc.

[Bogdan Dobrelya (bogdando)]

Actually, the impact of the open and unlinked files for logs free space consuming is not so HIGH, hence I update this status as meduim.

[Bogdan Dobrelya (bogdando)]

Superseded by https://bugs.launchpad.net/oslo/+bug/1276694

[Bogdan Dobrelya (bogdando)]

Introducing restarts for OSt services would be an ugly kludge, instead fixes for SIGHUP support (#1276694) should be be implemented by MOS-Oslo and perhaps contributed to Oslo as well.