[9.1][Scale][Timmy] Timmy return exit_code==0 even if error occured

[Maksym Strukov]

Steps:
0. Master node /var partition should be small, to run into 'not enough space' when collecting logs. On my env /var has ~50gb free space, and log snapshot need ~334gb
1. cd /var/dump
2. fuel snapshot

Actual result:

Fuel create empty log snapshot:

[root@fuel dump]# fuel snapshot
Generating dump...
Downloading: http://10.20.0.2:8000/api/dump/fuel-snapshot-2016-09-22_10-21-53.tar Bytes: 10240
[==============================================================================]()
[root@fuel dump]# ls -hal
total 20K
drwxr-xr-x. 2 root root 4.0K Sep 22 10:22 .
drwxr-xr-x. 25 root root 4.0K Sep 21 13:47 ..
-rw-r--r--. 1 root root 10K Sep 22 10:22 fuel-snapshot-2016-09-22_10-21-53.tar

I re-run timmy directly with same command as Fuel did:

[root@fuel dump]# timmy --logs --days 3 --dest-file /var/dump/fuel-snapshot-2016-09-22_10-21-53/config.tar.gz --log-file /var/log/timmy.log
Initializing node data: done
Calculating logs size: done
Total logs size to collect before compression: 334176MB.
Compressed logs will take less space.
Log collection period: 3 days.
Checking free space: done

And checking exit code:

[root@fuel dump]# echo $?
0

[root@fuel dump]# ls fuel-snapshot-2016-09-22_10-21-53/
...

2016-09-22 18:13:07,356 INFO: nodes: is_enough_space(): logsize: 334291MB * 1.05, free space: 57770MB
2016-09-22 18:13:07,356 ERROR: nodes: is_enough_space(): Not enough space in "/var/dump/fuel-snapshot-2016-09-22_10-21-53", logsize: 334291MB * 1.05, available: 57770MB. Decrease logs_size_coefficient config parameter (--logs-coeff CLI parameter) or free up space.
2016-09-22 18:13:07,356 ERROR: cli: main(): Not enough space for logs in "/var/dump/fuel-snapshot-2016-09-22_10-21-53", exiting.

Root problem is that Timmy return exit code 0 when out of free space and can't gather logs.
Fuel receive timmy exit code == 0 and creating empty tar file which return to User.

By default /var partition allocate 40% from master node HDD size

Workaround:
None

Environment:
[root@fuel dump]# timmy -V
1.20.3

Master node has HDD 500 gb with /var partition 185 GB
Cluster has 200 nodes, deployed more than 3 days ago

[Dmitry Sutyagin (dsutyagin)]

Thank you for the bug report!
Fix upstream - https://review.openstack.org/375052
Fix downstream - I hope we can create a review with an updated package tomorrow morning

[Dmitry Sutyagin (dsutyagin)]

Downstream review for an update package for 9.0 branch - https://review.fuel-infra.org/#/c/26797
For master branch - https://review.fuel-infra.org/26798

I hope tests will finish by tomorrow morning and we can get this approved and merged

[Vitaly Sedelnik (vsedelnik)]

Setting back to In Progress as https://review.fuel-infra.org/#/c/26797 is not merged. Also setting to Incomplete as it's not clear how this issue affects product.

Please update the description with steps to reproduce, actual and expected results. The steps have to deal with MOS 9.1 product, not just timmy package

[Dmitry Sutyagin (dsutyagin)]

Vitaly, would you please +1 this review so it gets merged?

[Dmitry Sutyagin (dsutyagin)]

Fixed upstream, merged into Mitaka & Newton