fuel-snapshot must contain kernel and system logs

[Pavel Boldin]

[Impact]
* Anyone who tries to debug any FUEL kernel- or system-related bug.

[Test Case]
HOW TO REPRODUCE
1. Ask Fuel to make a snapshot.
2. Download and look for kernel or system logs.

EXPECTED BEHAVIOUR
- There are kernel and system logs in the snapshot.

ACTUAL BEHAVIOUR
- None of the system-related logs are there.

[Dmitry Kalashnik (dkalashnik)]

Now it contains only masters' kern.log

>>>$ find . -name kern.log
./fuel-snapshot-2015-09-11_15-46-16/172.16.40.69/root/kern.log
./fuel-snapshot-2015-09-11_15-46-16/172.16.40.69/var/log/docker-logs/kern.log

[Dmitry Kalashnik (dkalashnik)]

[root@fuel-slab2 slab2_stab]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "288"
  build_id: "288"
  nailgun_sha: "93477f9b42c5a5e0506248659f40bebc9ac23943"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "a717657232721a7fafc67ff5e1c696c9dbeb0b95"
  fuel-library_sha: "121016a09b0e889994118aa3ea42fa67eabb8f25"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

[Dmitry Kalashnik (dkalashnik)]

The same for messages & syslog

>>>$ find . -name messages
./fuel-snapshot-2015-09-11_15-46-16/172.16.40.69/var/log/docker-logs/messages
./fuel-snapshot-2015-09-11_15-46-16/172.16.40.69/var/log/messages

>>>$ find . -name syslog
./fuel-snapshot-2015-09-11_15-46-16/172.16.40.69/var/log/docker-logs/syslog

[Vladimir Kuklin (vkuklin)]

Folks

This bug is invalid. All logs sent through system log goes into debug snapshot for slave nodes. Please look into docker-logs/remote/<node-hostname>/kernel.log

[Pavel Boldin (pboldin)]

This bug is valid.

I want a full kern.log file, not a slice of it. Please fix the snapshot utility.

kernel.log starts with:
2015-09-08T15:18:56.509085+00:00 info: [ 334.610535] gre: GRE over IPv4 demultiplexor driver

[Maciej Kwiek (maciej-iai)]

I can see that messages.log are included in docker-logs/remote lgos. Will check for kernel and syslog.

[Maciej Kwiek (maciej-iai)]

Shotgun seems to not save /var/log/docker-logs/remote into the snapshot right now. Investigating.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/225660

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/225660
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=2a327bd01e4e8570c28f10ce4a205b31868f5056
Submitter: Jenkins
Branch: master

commit 2a327bd01e4e8570c28f10ce4a205b31868f5056
Author: Maciej Kwiek <email address hidden>
Date: Tue Sep 29 09:50:10 2015 +0200

    Remove dangling symlinks after node log removal

    Dangling symlinks caused snapshots to generate without part of logs. The
    symlinks were a leftover after deleteing logs from deleted nodes.

    Existence of log folders was checked using os.path.exists function, it
    was changed to os.path.lexists, which returns True for broken symlinks.

    Change-Id: I446df0595ba668800e89f378ab08bd33cec4fb56
    Closes-Bug: #1494838

[Vitaly Sedelnik (vsedelnik)]

Maciej, please review the fix and backport to 7.0 if applicable. If not, please update the status accordingly (Won't Fix or Invalid).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/232558

[Ksenia Svechnikova (kdemina)]

Should be verified when issue https://bugs.launchpad.net/fuel/+bug/1505180 will be fixed

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/232558
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=50c25db9de0b11c8730a9d2c975bb4770258ab2b
Submitter: Jenkins
Branch: stable/7.0

commit 50c25db9de0b11c8730a9d2c975bb4770258ab2b
Author: Maciej Kwiek <email address hidden>
Date: Tue Sep 29 09:50:10 2015 +0200

    Remove dangling symlinks after node log removal

    Dangling symlinks caused snapshots to generate without part of logs. The
    symlinks were a leftover after deleteing logs from deleted nodes.

    Existence of log folders was checked using os.path.exists function, it
    was changed to os.path.lexists, which returns True for broken symlinks.

    Change-Id: I446df0595ba668800e89f378ab08bd33cec4fb56
    Closes-Bug: #1494838
    (cherry picked from commit 2a327bd01e4e8570c28f10ce4a205b31868f5056)

[Andrey Sledzinskiy (asledzinskiy)]

7.0 fix was verified on 7.0-301 release iso with MU updates

[Nastya Urlapova (aurlapova)]

Moved to Fix Released according to a comment above.

[Andrey Lavrentyev (alavrentyev)]

8.0 has been verified on ISO #168

# fuel fuel-version
api: '1.0'
astute_sha: 959b06c5ef8143125efd1727d350c050a922eb12
auth_required: true
build_id: '168'
build_number: '168'
feature_groups:
- mirantis
fuel-agent_sha: 2750600e946e96701099dfef5a7d69017f2e9956
fuel-createmirror_sha: e34a0dd080fe6c133bcc75a00c31b27934ba51bc
fuel-library_sha: 0d210dc3851ac6c74b3914ef4df0139defc8d117
fuel-nailgun-agent_sha: 3e9d17211d65c80bf97c8d83979979f6c7feb687
fuel-nailgun_sha: a5f4c44d08715f3be6b200a69032b30d347ac911
fuel-ostf_sha: 9690a2de829d3b063ed1e64b0b10dde39f711dc0
fuel-upgrade_sha: 1e894e26d4e1423a9b0d66abd6a79505f4175ff6
fuelmain_sha: 266f9b374934c64629a84b5146632cc0de70ac91
fuelmenu_sha: 06bbcebed6c8d0b0f9279e2997d2f958c800e98c
network-checker_sha: a57e1d69acb5e765eb22cab0251c589cd76f51da
openstack_version: 2015.1.0-8.0
production: docker
python-fuelclient_sha: e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c
release: '8.0'
release_versions:
  2015.1.0-8.0:
    VERSION:
      api: '1.0'
      astute_sha: 959b06c5ef8143125efd1727d350c050a922eb12
      build_id: '168'
      build_number: '168'
      feature_groups:
      - mirantis
      fuel-agent_sha: 2750600e946e96701099dfef5a7d69017f2e9956
      fuel-createmirror_sha: e34a0dd080fe6c133bcc75a00c31b27934ba51bc
      fuel-library_sha: 0d210dc3851ac6c74b3914ef4df0139defc8d117
      fuel-nailgun-agent_sha: 3e9d17211d65c80bf97c8d83979979f6c7feb687
      fuel-nailgun_sha: a5f4c44d08715f3be6b200a69032b30d347ac911
      fuel-ostf_sha: 9690a2de829d3b063ed1e64b0b10dde39f711dc0
      fuel-upgrade_sha: 1e894e26d4e1423a9b0d66abd6a79505f4175ff6
      fuelmain_sha: 266f9b374934c64629a84b5146632cc0de70ac91
      fuelmenu_sha: 06bbcebed6c8d0b0f9279e2997d2f958c800e98c
      network-checker_sha: a57e1d69acb5e765eb22cab0251c589cd76f51da
      openstack_version: 2015.1.0-8.0
      production: docker
      python-fuelclient_sha: e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c
      release: '8.0'
      shotgun_sha: 25dd78a3118267e3616df0727ce746e7dead2d67
shotgun_sha: 25dd78a3118267e3616df0727ce746e7dead2d67

[Anastasia Palkina (apalkina)]

Also verified on logs for ISO #284