fuel creates two diagnostic snapshots instead of one

[Leontii Istomin]

[root@c9b86c3524db /]# fuel --fuel-version
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-01-21_20-49-45
build_number: '41'
feature_groups:
- mirantis
fuellib_sha: 9aa913096fb93ea4847ee14bfaf33597326886f3
fuelmain_sha: 1ee1766a51bdb5bed75d5c2efdcaaa318118e439
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: 3d2f44dcfa32d6ce0372cc64695e9edcc1913ea7
production: docker
release: 6.0.1
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
      build_id: 2015-01-21_20-49-45
      build_number: '41'
      feature_groups:
      - mirantis
      fuellib_sha: 9aa913096fb93ea4847ee14bfaf33597326886f3
      fuelmain_sha: 1ee1766a51bdb5bed75d5c2efdcaaa318118e439
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: 3d2f44dcfa32d6ce0372cc64695e9edcc1913ea7
      production: docker
      release: 6.0.1

Centos+HA+Neutron-gre+Ceilometer+LVM+Debug+6.0.1_41
Controllers:3 Computes:95

I generate diagnostic snapshot via UI, since some time I can see on fuel node:

[root@fuel lib]# du -hs /var/www/nailgun/dump/*
4.6G /var/www/nailgun/dump/fuel-snapshot-2015-01-23_14-24-49.tgz
5.4G /var/www/nailgun/dump/fuel-snapshot-2015-01-23_15-24-54
4.0K /var/www/nailgun/dump/last

[root@fuel lib]# fuel task
id | status | name | cluster | progress | uuid
---|---------|----------------|---------|----------|-------------------------------------
1 | ready | check_networks | 1 | 100 | c853c5f2-4e7f-468c-8eae-b21b3dd68ce1
7 | running | dump | None | 0 | efda081b-86a0-4bfe-bfc4-b80f1a8f775a

[root@c9b86c3524db /]# grep snapshot /var/log/nailgun/app.log
2015-01-23 14:24:48.293 DEBUG [7f71f1555740] (task) Preparing config for snapshot
2015-01-23 14:24:48.632 DEBUG [7f71f1555740] (task) Dump conf: {'timestamp': True, 'lastdump': '/var/www/nailgun/dump/last', 'target': '/var/www/nailgun/dump/fuel-snapshot', 'dump': {'master': {'objects': [{'path': '/etc/nailgun', 'type': 'dir'}, {'path': '/etc/astute', 'type': 'dir'}, {'path': '/etc/fuel', 'type': 'dir'}, {'command': 'df -h', 'to_file': 'df.txt', 'type': 'command'}, {'command': 'mount', 'to_file': 'mount.txt', 'type': 'command'}, {'command': 'iptables -t nat -S', 'to_file': 'iptables_nat.txt', 'type': 'command'}, {'command': 'iptables -t filter -S', 'to_file': 'iptables_filter.txt', 'type': 'command'}, {'command': 'dmidecode', 'to_file': 'dmidecode.txt', 'type': 'command'}, {'command': 'uptime', 'to_file': 'uptime.txt', 'type': 'command'}, {'command': 'ps auxwwf', 'to_file': 'ps.txt', 'type': 'command'}, {'command': 'pvdisplay', 'to_file': 'lvm_pvdisplay.txt', 'type': 'command'}, {'command': 'docker images', 'to_file': 'docker_images.txt', 'type': 'command'}, {'command': 'docker ps', 'to_file': 'docker_ps.txt', 'type': 'command'}, {'command': 'vgdisplay', 'to_file': 'lvm_vgdisplay.txt', 'type': 'command'}, {'command': 'lvdisplay', 'to_file': 'lvm_lvdisplay.txt', 'type': 'command'}, {'command': 'ip a', 'to_file': 'ip_a.txt', 'type': 'command'}, {'command': 'ip r', 'to_file': 'ip_r.txt', 'type': 'command'}, {'command': 'netstat -anp', 'to_file': 'netstat.txt', 'type': 'command'}, {'command': 'brctl show', 'to_file': 'brctl_show.txt', 'type': 'command'}, {'path': '/etc/sysconfig/network-scripts', 'type': 'dir'}, {'path': '/etc/dnsmasq.conf', 'type': 'file'}, {'path': '/root/*.log', 'type': 'file'}, {'path': '/etc/cobbler*', 'type': 'file'}, {'path': '/var/log', 'type': 'dir'}], 'hosts': [{'ssh-key': '/root/.ssh/id_rsa', 'address': '10.20.0.2'}]}, 'controller': {'objects': [{'command': 'pcs status', 'to_file': 'pcs_status.txt', 'type': 'command'}, {'command': 'rabbitmqctl report', 'to_file': 'rabbitmqctl_report.txt', 'type': 'command'}], 'hosts': [{'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-21.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-96.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-91.domain.tld'}]}, 'local': {'objects': [{'username': 'nailgun', 'dbhost': '10.20.0.2', 'password': '6Qri3GjE', 'type': 'postgres', 'dbname': 'nailgun'}, {'to_file': 'cobbler.txt', 'type': 'xmlrpc', 'methods': ['get_distros', 'get_profiles', 'get_systems'], 'server': 'http://10.20.0.2:80/cobbler_api'}], 'hosts': [{'ssh-key': '/root/.ssh/id_rsa', 'address': 'localhost'}]}, 'slave': {'objects': [{'path': '/etc/astute.yaml', 'type': 'file'}, {'path': '/var/log/puppet.log', 'type': 'file'}, {'path': '/root/ceph*', 'type': 'file'}, {'path': '/root/anaconda*', 'type': 'file'}, {'path': '/root/*.log', 'type': 'file'}, {'path': '/root/*.ks', 'type': 'file'}, {'path': '/etc/ceph*', 'type': 'file'}, {'path': '/etc/keystone*', 'type': 'file'}, {'path': '/etc/nova*', 'type': 'file'}, {'path': '/etc/neutron*', 'type': 'file'}, {'path': '/etc/horizon*', 'type': 'file'}, {'path': '/etc/cinder*', 'type': 'file'}, {'path': '/etc/glance*', 'type': 'file'}, {'path': '/etc/swift*', 'type': 'file'}, {'path': '/etc/haproxy*', 'type': 'file'}, {'path': '/var/log/ceph', 'type': 'file'}, {'path': '/var/lib/puppet/state/last_run_summary.yaml', 'type': 'file'}, {'path': '/etc/resolv.conf', 'type': 'file'}, {'path': '/var/run/pcap_dir', 'type': 'dir'}, {'path': '/var/log/atop/atop_current', 'type': 'file'}, {'command': 'df -h', 'to_file': 'df.txt', 'type': 'command'}, {'command': 'mount', 'to_file': 'mount.txt', 'type': 'command'}, {'command': 'iptables -t nat -S', 'to_file': 'iptables_nat.txt', 'type': 'command'}, {'command': 'iptables -t filter -S', 'to_file': 'iptables_filter.txt', 'type': 'command'}, {'command': 'dmidecode', 'to_file': 'dmidecode.txt', 'type': 'command'}, {'command': 'uptime', 'to_file': 'uptime.txt', 'type': 'command'}, {'command': 'ps auxwwf', 'to_file': 'ps.txt', 'type': 'command'}, {'command': 'pvdisplay', 'to_file': 'lvm_pvdisplay.txt', 'type': 'command'}, {'command': 'vgdisplay', 'to_file': 'lvm_vgdisplay.txt', 'type': 'command'}, {'command': 'lvdisplay', 'to_file': 'lvm_lvdisplay.txt', 'type': 'command'}, {'command': 'ip a', 'to_file': 'ip_a.txt', 'type': 'command'}, {'command': 'ip r', 'to_file': 'ip_r.txt', 'type': 'command'}, {'command': 'netstat -anp', 'to_file': 'netstat.txt', 'type': 'command'}, {'command': 'brctl show', 'to_file': 'brctl_show.txt', 'type': 'command'}, {'path': '/etc/sysconfig/network-scripts', 'type': 'dir'}, {'path': '/etc/network/interfaces.d', 'type': 'dir'}, {'path': '/etc/network/interfaces', 'type': 'file'}, {'path': '/root/post-partition.log', 'type': 'file'}, {'command': 'blkid -o list', 'to_file': 'blkid_o_list.txt', 'type': 'command'}, {'command': 'ceph health', 'to_file': 'ceph_health.txt', 'type': 'command'}, {'command': 'ceph -s', 'to_file': 'ceph_s.txt', 'type': 'command'}, {'command': 'ceph osd tree', 'to_file': 'ceph_osd_tree.txt', 'type': 'command'}, {'command': 'cat /proc/interrupts', 'to_file': 'proc_interrupts.txt', 'type': 'command'}], 'hosts': [{'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-68.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-89.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-81.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-65.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-13.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-10.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-92.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-6.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-27.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-24.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-62.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-71.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-3.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-5.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-63.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-55.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-84.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-79.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-41.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-83.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-7.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-80.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-29.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-59.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-21.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-70.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-54.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-72.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-75.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-73.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-35.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-93.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-87.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-31.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-86.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-46.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-95.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-25.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-53.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-42.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-14.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-32.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-77.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-16.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-28.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-78.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-43.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-45.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-57.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-47.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-23.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-76.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-48.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-49.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-20.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-94.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-96.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-22.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-4.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-88.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-19.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-18.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-82.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-64.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-36.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-12.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-1.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-2.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-9.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-8.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-11.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-15.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-17.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-34.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-33.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-30.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-39.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-40.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-37.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-50.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-44.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-51.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-52.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-58.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-60.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-69.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-66.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-67.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-74.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-85.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-90.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-91.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-97.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-98.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-61.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-26.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-38.domain.tld'}, {'ssh-key': '/root/.ssh/id_rsa', 'address': u'node-56.domain.tld'}]}}}
            "target": "/var/www/nailgun/dump/fuel-snapshot",

[Leontii Istomin (listomin)]

I can reproduce this behavior on each configuration.

[Aleksey Kasatkin (alekseyk-ru)]

More info from Leontiy:

file list:
4.6G /var/www/nailgun/dump/fuel-snapshot-2015-01-23_14-24-49.tgz
5.4G /var/www/nailgun/dump/fuel-snapshot-2015-01-23_15-24-54
4.0K /var/www/nailgun/dump/last

from log:
[root@c9b86c3524db /]# grep snapshot /var/log/nailgun/app.log
2015-01-23 14:24:48.293 DEBUG [7f71f1555740] (task) Preparing config for snapshot
2015-01-23 14:24:48.632 DEBUG [7f71f1555740] (task) Dump conf: {...} "target": "/var/www/nailgun/dump/fuel-snapshot",...

It seems that timestamp difference (15-24-54 - 14-24-49) is the duration of log compression.

[Ryan Moe (rmoe)]

Leontiy, can you also provide the /var/log/docker-logs/dump.log file?

[Leontii Istomin (listomin)]

/var/log/docker-logs/dump.log file is empty

[Dima Shulyak (dshulyak)]

Guys, it is different snapshots, we are keeping not only last one, but all snapshot that are created

[root@nailgun dump]# ls -lh
total 298M
-rw-r--r-- 1 root root 23M Feb 6 15:16 fuel-snapshot-2015-02-06_15-16-14.tgz
-rw-r--r-- 1 root root 24M Feb 6 15:26 fuel-snapshot-2015-02-06_15-25-27.tgz
-rw-r--r-- 1 root root 252M Feb 9 09:25 fuel-snapshot-2015-02-09_09-24-08.tgz
-rw-r--r-- 1 root root 59 Feb 9 09:25 last

I am not sure that it is ok to remove previous snapshots, there maybe information that is not available in new one, due to log rotation

[Leontii Istomin (listomin)]

Please pay attention, that I initialized generating of diagnostic snapshot only once in the installation. I don't mean that we need to delete old snapshots. But I mean that when we want to generate one diagnostic snapshot, we get more then one. Please note, that there is 1 hour between snapshots.
4.6G /var/www/nailgun/dump/fuel-snapshot-2015-01-23_14-24-49.tgz
5.4G /var/www/nailgun/dump/fuel-snapshot-2015-01-23_15-24-54

Also the second snapshot is bigger the first one. Probably, we have some timeout for snapshotting?

[Dima Shulyak (dshulyak)]

So it happens because snapshot generation takes more time than it is expected (60 min).

https://github.com/stackforge/fuel-astute/blob/master/lib/astute/dump.rb#L21

As a quick fix we may increase timeout, but it is not proper solution, and will help only in very limited way

We need to address this problem in several ways:

1. Reduce logs amount
2. Reduce time of compression

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/158237

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/158266

[Dima Shulyak (dshulyak)]

Assigning back to fuel-python, will get back to them ater ff

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/158266
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=fa3aa37547359d238f8a61fd980ea0645fa8c2a9
Submitter: Jenkins
Branch: master

commit fa3aa37547359d238f8a61fd980ea0645fa8c2a9
Author: Dmitry Shulyak <email address hidden>
Date: Mon Feb 23 13:51:39 2015 +0200

    Allow to provide timeout for snapshot generation

    removed automatic retries, they can cause duplication of tarballs
    even when snapshot can not be generated in given timeout

    Change-Id: I4c5fec5d72d2ed76528d1b03d878ca0572e012f2
    Closes-Bug: 1414059

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/158237
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=4eccce774023dd31071d72fe3eca697f644efd3b
Submitter: Jenkins
Branch: master

commit 4eccce774023dd31071d72fe3eca697f644efd3b
Author: Dmitry Shulyak <email address hidden>
Date: Mon Feb 23 12:36:00 2015 +0200

    Allow to overwrite timeout for snapshot generation

    timeout variable specified in seconds and by default would be 3600 sec

    Closes-Bug: 1414059
    DocImpact

    Change-Id: I7e3df148b81fcfcc27a655675ed7582515976bfb

[Dina Belova (dbelova)]

Verified on the #233 ISO (6.1)