Fuel doesn't cleanup logs in /var/log/remote/ when environment is deleted or node removed

[Alexander Bozhenko]

When fuel installation is used for tests and environments are created and deleted frequently fuel keeps log dirs for deleted nodes in /var/log/remote forever.
It cause exhausting of free space on root partition very quickly.
You can propose me to propose customer just manually remove logs, BUT! in mox team edition they do not have root access, and can't do it.

I think fuel should either remove logs along with removing environment silently, or ask user about doing this.

104M /var/log/remote/node-138.domain.tld
107M /var/log/remote/node-130.domain.tld
127M /var/log/remote/node-101.domain.tld
132M /var/log/remote/node-109.domain.tld
137M /var/log/remote/node-19.domain.tld
161M /var/log/remote/node-102.domain.tld
243M /var/log/remote/192.168.115.201
243M /var/log/remote/192.168.121.53
243M /var/log/remote/192.168.92.196
243M /var/log/remote/192.168.93.250
245M /var/log/remote/192.168.105.219
245M /var/log/remote/192.168.93.28
245M /var/log/remote/192.168.93.30
273M /var/log/remote/node-151.domain.tld
274M /var/log/remote/node-152.domain.tld
400M /var/log/remote/node-83.domain.tld
408M /var/log/remote/node-82.domain.tld
456M /var/log/remote/node-118.domain.tld
456M /var/log/remote/node-126.domain.tld
545M /var/log/remote/node-150.domain.tld
556M /var/log/remote/node-116.domain.tld
559M /var/log/remote/node-115.domain.tld
710M /var/log/remote/node-113.domain.tld
714M /var/log/remote/node-114.domain.tld
826M /var/log/remote/node-112.domain.tld
836M /var/log/remote/node-145.domain.tld
847M /var/log/remote/node-146.domain.tld
1.5G /var/log/remote/node-144.domain.tld
15G total

[Łukasz Oleś (loles)]

I think it should ask user. Silently deleting logs may lead to problems

[Evgeniy L (rustyrobot)]

+1 to Lukasz, logs is a history, I don't think that we should delete them, but probably what we can do is to archive logs.

[Alexander Bozhenko (alexbozhenko)]

Do we really need history for nonexistent environment?
BTW, all these old logs included to snapshot, and we definitely do not need them there.

[Dmitry Pyzhov (dpyzhov)]

Logs should not be blindly deleted. Logrotate should remove old logs or something like this.

[Alexander Bozhenko (alexbozhenko)]

Dmitriy, why incomplete? It is a customer-found bug.
Is description not accurate "Fuel doesn't cleanup logs in /var/log/remote/ when environment is deleted"?
Is it not true, or what?

You wrote:
>>Logrotate should remove old logs or something like this.
That means our current logrotate configuration contains a bug.

[Dmitry Pyzhov (dpyzhov)]

Alexander, don't worry. I've moved it to Incomplete in order to highlight that I need your response. Also, I've discussed this bug with Matthew Mosesohn

So. First of all, we do rotate logs. If it is ok for you - let's close the bug as Won't fix. Logs from your old environments will be truncated at some point.

If log rotation is broken - please reopen the bug.

If you don't think that log rotation is an answer here - let's continue discussion and maybe move to the mailing list.

[Alexander Bozhenko (alexbozhenko)]

We have these logrotate configuration.
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/templates/10-fuel-docker.conf.erb
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/templates/20-fuel-docker.conf.erb
As far as I understand, nothing there can delete logs from old environments.

[Alexander Bozhenko (alexbozhenko)]

So, I think when removing a node we need:
1) Inform user in fuel web interface(and cli client help) what will happen
2) Remove corresponding directory and symlink from /var/log/remote

When removing whole environment remove_nodes method for each node will be executed, and all old logs will be removed.

After deleting node it is gone: it is gone from cobbler, postgres and it's disk is wiped out, there is no reason to keep logs on fuel master node.

[Bogdan Dobrelya (bogdando)]

Logrotate cannot ensure such logic, this should be done at the Nailgun/shotgun backend side

[Alexander Bozhenko (alexbozhenko)]

I think that it will be good to add note to the UI about what fuel is going to do when deleting a node.
It recently was improved here:
https://bugs.launchpad.net/fuel/+bug/1442335

[Sylwester Brzeczkowski (sbrzeczkowski)]

https://review.openstack.org/#/c/174879/

[Nikolay Markov (nmarkov)]

bogdando, strongly disagree. Logrotate and compressing should be used in case of node removal, not complete deletion.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/174879
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=4c8dce98d7719f29ba16a64be5edac6d970e0e6a
Submitter: Jenkins
Branch: master

commit 4c8dce98d7719f29ba16a64be5edac6d970e0e6a
Author: Sylwester Brzeczkowski <email address hidden>
Date: Fri Apr 17 15:42:01 2015 +0200

    Delete logs on master after node or env deletion

    Added delete_node_logs function to utils which is called in
    NailgunReciever.remove_nodes_resp after node deletion
    (just before removing it from database).

    Closes-Bug: #1428825
    Change-Id: I828e515c7f5d1e39373d351809e583cd732836c1

[Anastasia Palkina (apalkina)]

Verified on ISO #408

"build_id": "2015-05-12_17-03-01", "build_number": "408", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-12_17-03-01", "build_number": "408", "api": "1.0", "fuel-library_sha": "156fb11bbf3e12e7c73a9a3ac785c9d33d4ac343", "nailgun_sha": "042ed77ffdff22a5242ac7f8a6980836d0a5ef1a", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "af6c9c3799b9ec107bcdc6dbf035cafc034526ce", "astute_sha": "d9d488c4c675e6dd33eb68b33a79abe591b4c26f", "fuel-ostf_sha": "21afa436e725be1debadf1c207018753b537c7b3", "release": "6.1", "fuelmain_sha": "51b86bb24b27742a22b23e2ae3dfc850c47e5fbf"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "156fb11bbf3e12e7c73a9a3ac785c9d33d4ac343", "nailgun_sha": "042ed77ffdff22a5242ac7f8a6980836d0a5ef1a", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "af6c9c3799b9ec107bcdc6dbf035cafc034526ce", "astute_sha": "d9d488c4c675e6dd33eb68b33a79abe591b4c26f", "fuel-ostf_sha": "21afa436e725be1debadf1c207018753b537c7b3", "release": "6.1", "fuelmain_sha": "51b86bb24b27742a22b23e2ae3dfc850c47e5fbf"

[Dmitry Borodaenko (angdraug)]

This is a High priority customer-found bug, status for 6.0.x changed from Won't Fix to Confirmed.

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 6.0-updates as this is feature request not a bug