Generating a diagnostic snapshot triggers to an error "exit code:1 stderr:"

[Sergii Turivnyi]

Steps to reproduce:
Deploy environment 3 controllers, 2 ironic+ceph
Active use it for 2 weeks
Generate Diagnostic snapshot

Expected result:
Snapshot was generated successfully

Actual result:
Generating a diagnostic snapshot triggers to an error "exit code:1 stderr:" (see attachment)

Logs:

[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 8,2G 859M 91% /var
/dev/mapper/os-varlog 101G 8,7G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 8,8G 223M 98% /var
/dev/mapper/os-varlog 101G 8,8G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 8,9G 86M 100% /var
/dev/mapper/os-varlog 101G 8,8G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 9,0G 37M 100% /var
/dev/mapper/os-varlog 101G 8,8G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]#

Possible way to fix: save logs to /dev/mapper/os-varlog

[Nastya Urlapova (aurlapova)]

@Sergey, what Fuel version did you use?

[Sergii Turivnyi (sturivnyi)]

{u'release': u'8.0', u'api': u'1', u'openstack_version': u'liberty-8.0', u'feature_groups': [u'mirantis'], u'auth_required': True}

[Georgy Kibardin (gkibardin)]

As usual there are to solutions: fast to implement and more optimal from storage space handling point of view.
The first, a fastest solution would be just using big enough partition for the whole /var (not /var/log - not sure what practical outcome this decision brings)
The second, more optimal solution, would be to compress snapshot data on nodes and then "tar" on the master node when necessary on the fly during downloading it via Fuel UI. This would free a lot of space required to store uncompressed data.

[Georgy Kibardin (gkibardin)]

The second part can be done in bounds of https://blueprints.launchpad.net/fuel/+spec/manage-logs-with-free-space-consideration

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/310449

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/310452

[OpenStack Infra (hudson-openstack)]

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: master
Review: https://review.openstack.org/310452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310449
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=410e0b9c23020aaf977e699d2d643036193092b7
Submitter: Jenkins
Branch: master

commit 410e0b9c23020aaf977e699d2d643036193092b7
Author: Georgy Kibardin <email address hidden>
Date: Wed Apr 27 15:37:01 2016 +0300

    Don't make a copy of master node logs

    Master node host creds are injected into shotgun config making it
    connect to localhost via ssh and make a copy of logs in /var/dump.
    Without creds, which aren't necessary anymore since we don't use
    containers, it just makes a symlink saving a lot of space.
    Also fixes incorrect excludes.

    Change-Id: I2e1a60bd71955b2c607f46d2cde60daa49811be7
    Partial-Bug: #1543491

[Georgy Kibardin (gkibardin)]

Since we are going to deprecate shotgun we've decided not to fix space issue. At least, because it is not shotgun issue.
The fix only improves diagnostic so that we know what the problem is.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/322756

[Sofiia Andriichenko (sandriichenko)]

[root@nailgun ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 2.0G 7.1G 22% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 183M 1.7G 10% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vda3 197M 129M 68M 66% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 20G 3.8G 16G 20% /var
/dev/mapper/os-varlog 31G 1.9G 27G 7% /var/log
tmpfs 380M 0 380M 0% /run/user/0
tmpfs 380M 0 380M 0% /run/user/493
[root@nailgun ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 2.0G 7.1G 22% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 183M 1.7G 10% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vda3 197M 129M 68M 66% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 20G 5.9G 14G 31% /var
/dev/mapper/os-varlog 31G 1.9G 27G 7% /var/log
tmpfs 380M 0 380M 0% /run/user/0
tmpfs 380M 0 380M 0% /run/user/493

verified on:
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 465
cat /etc/fuel_build_number:
 465
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8454.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8454.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-openstack-metadata-9.0.0-1.mos8742.noarch
 fuel-notify-9.0.0-1.mos8454.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8454.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8742.noarch
 fuel-library9.0-9.0.0-1.mos8454.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos935.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8742.noarch