Shotgun makes a second copy of a /var/log/remote while creating a snapshot

[Alexander Bozhenko]

While creating a snapshot shotgun:
1) creates a second copy of a /var/log/remote to /var/www/nailgun/dump/fuel-snapshot-$date-$time
2) archive created directory using tar
3) and delete the directory

So, to successfully get a snapshot you need to have free space on the disk at least double amount of /var/log/remote.

We could try to create a list of files to include, and utilize -T option of tar:
       -T, --files-from=FILE
              get names to extract or create from FILE

It will reduce time for snapshot and mitigate free space requirements.

[Alexander Bozhenko (alexbozhenko)]

Sorry, it is for fuel, not for mos.

[Tomasz 'Zen' Napierala (tzn)]

Alexander,
This does not comply with "High" definition for Fuel, moreover it's more feature request, not a bug. I understand that from operational perspective it's pain for customers, but we need to lower the priority. Also because it need redesign, I need to move it to next release

[Alexander Bozhenko (alexbozhenko)]

Although, I agree that it requires redesign and we already in SCF for 6.1, but let's make sure that in half an year this will not happen:
milestone: 7.0 → 7.1

[Alexander Kislitsky (akislitsky)]

Feature not in progress. Moving status to Confirmed

[Vladimir Sharshov (vsharshov)]

Moving to 8.0, because it is feature.

[Dmitry Mescheryakov (dmitrymex)]

Did a little magic to help lp-reports consume the bug. The bug status is not changed.

[Alexander Gordeev (a-gordeev)]

To reduce the overhead brought by 'cp' call, we could recreate directory structure from scratch and add symlinks to all files. Also, we should pass '-h' option to tar.

Does it still need bp? I don't think so. It sounds just as an ordinary minor improvement.

I doubt the necessity of managing of a list of files consumed by tar.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/231620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/231621

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/231620
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=c43e2fe1d256a2a428ba247cda2fadfb3d506cde
Submitter: Jenkins
Branch: master

commit c43e2fe1d256a2a428ba247cda2fadfb3d506cde
Author: Alexander Gordeev <email address hidden>
Date: Fri Oct 2 18:27:40 2015 +0300

    Ditch Subs

    Initially, Subs was introduced to substitute sensitive data from logs
    which are getting included into diagnostic snapshot.
    However, it was turned off by default since fuel 5.1

    And we're in a situation where this piece of code was left untouched
    for almost 2 year or so.

    By getting rid of it, we is going to solve few issues:
    1) devs will no longer need to spend time to maintain unused piece of
    code.
    2) shotgun will stop to modify the actual data inside of log
    files. Therefore, this will allow us to significally reduce waste of
    disk space and i/o on preparing copy of locally included logs.

    Change-Id: Id78e369ea04e2847025c15150038de5f508a33af
    Related-Bug: #1445311

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/239495

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandr Gordeev (<email address hidden>) on branch: master
Review: https://review.openstack.org/231621
Reason: abandoned, backported to new repo at openstack/shotgun

[Matthew Mosesohn (raytrac3r)]

I had a discussion with Alexey. We proposed making the Fuel Master host system /var/log accessible directly inside the mcollective container, so that shotgun can access it more directly. To do this we should add the following lines in dockerctl config:

HOST_VOL['hostlogs']="-v /var/log:/log"

REQS["${HOST_VOL['hostlogs']}"]="mcollective"

This would allow us to make symlinks when copying files from a local source to reduce disk space requirements and use tar 'h' option to back up symlinks as files.

With regard to /var/log/docker-logs:/var/log references, we should keep that mapping in place so that IBP and normal mcollective tasks work correctly.

[Bug Checker Bot (esikachev-l)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Roman Prykhodchenko (romcheg)]

Moving this feature bug to Fuel Newton.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/239495
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.