Not a whole /var/log/ is used for diagnostic snapshot

[Mike Scherbakov]

Current implementation (see https://review.openstack.org/#/c/93178/) assumes that developers explicitly provide log files to be included into diagnostic snapshot archive. I consider this to be a very bad practice, when system relies on memory of developers. We’ve already forgotten about a several important log files, and I envision even more issues like this in the future (log file renamed, but config is not updated - so log doesn’t get included into snapshot, etc.).
It has to be reimplemented in a way when the whole folder gets included. If we need to, we can introduce a list of excludes.

[Evgeniy L (rustyrobot)]

I tried to add entire /var/log directory, but for some reasons it fails

Warning: get() encountered an exception while downloading '/var/log/'

Underlying exception:
    Failure

I didn't have enough time to debug it
Here is config [1] which you can use for debugging, use it in mcollective container like `shotgun -c /tmp/cfg`

[1] http://paste.openstack.org/show/84386/

[Ihor Kalnytskyi (ikalnytskyi)]

Well, the issue is in shotgun's engine - it can't work with symlinks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/102242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/102242
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=9be3661d726f57c5aa574a0b2b7b87d05677f3af
Submitter: Jenkins
Branch: master

commit 9be3661d726f57c5aa574a0b2b7b87d05677f3af
Author: Igor Kalnitsky <email address hidden>
Date: Tue Jun 24 15:05:44 2014 +0300

    Shotgun: make snapshot of entire /var/log

    Shootgun uses a fabric.api.get method to download some folder from the
    remote host. Unfortunately, this method has a bug and fails when we
    trying to download a folder that contains a symlink to another one.

    So we need to patch fabric api to prevent us from such falls. The issue
    with fabric api is that it recognize all symlinks as a file. As a result
    we fails when trying to download a folder (pointed by a symlink) as
    a file.

    The patch was proposed to the fabric master branch and can be reviewed
    here:

        https://github.com/fabric/fabric/pull/1147

    Closes-Bug: #1318514
    Closes-Bug: #1323436

    Change-Id: Ic2ae889337e5da15bbdf83338b8e8d5d913aeb63

[Dmitry Borodaenko (angdraug)]

Marking as Won't Fix for 5.0.1, please propose backport if you think impact of this change is small enough for a maintenance release.

[Meg McRoberts (dreidellhasa)]

Marked as Known Issue in 5.0.1 Release Notes.

[Dmitry Teselkin (teselkin-d)]

Diagnostic snapshot still incomplete - there are no logs from slaves in snapshot.
For example, https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/1749/

[Dmitry Teselkin (teselkin-d)]

Update - logs from slave nodes seems to be stored in 'docker-logs/remote' directory. However, files that could be found there is not enough. At the moment there are only 4 files (bootstrap):
* agent.log
* dmesg.log
* mcollective.log
* messages.log

It's not enough, need at least the following files to be added:
* auth.log
* syslog

An output of command 'ps axjf' also should be logged to show the processes running on the node.

[Dmitry Pyzhov (dpyzhov)]

Actually, this request is fixed. All logs from /var/log are in snapshot. Issue with logs from docker-logs/remote should be reported as a separate issue