Systest log transforms yum output and it became unreadable

Bug #1585194 reported by Igor Yozhikov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Medium
Igor Yozhikov

Bug Description

Detailed bug description:
Yum update log doesn't represented as it was produced! To find issue in this log it is required to extract related to package manager part and transform locally(substitue '\n' to real new line)

Log:
https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-centos/578/artifact/fuel-qa/logs/sys_test.log/*view*/

Syslog test log output:
...
2016-05-24 10:22:21,325 - DEBUG decorators.py:242 -- Result of "yum update" command on master node: {'exit_code': 0, 'stderr': [], 'stdout': ['Loaded plugins: fastestmirror, priorities\n', 'Cleaning repos: mos mos10.0-security mos10.0-updates nailgun temporary\n', '2 metadata files removed\n', 'Loaded plugins: fastestmirror, priorities\n', 'Config time: 0.010\n', 'Yum version: 3.4.3\n', 'rpmdb time: 0.000\n', 'Building updates object\n', 'Setting up Package Sacks\n', 'Loading mirror speeds from cached hostfile\n', ' * mos10.0-security: mirror.seed-us1.fuel-infra.org\n', ' * mos10.0-updates: mirror.seed-us1.fuel-infra.org\n', ' --> cloud-init-0.7.5-6.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> cloud-utils-0.27-10.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> cloud-utils-growpart-0.27-10.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> gperftools-2.4-5.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> gperftools-devel-2.4-5.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> gperftools-libs-2.4-5.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> libunwind-1.1-10.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> pprof-2.4-5.el7~mos1.noarch from mos excluded (priority)\n', ' --> python-cherrypy-3.2.2-4.el7~mos2.noarch from mos excluded (priority)\n', ' --> python-crypto-2.6.1-1.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> python-docker-py-1.3.1-1.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> python-enum34-1.0-4.el7~mos1.noarch from mos excluded (priority)\n', ' --> 1:python-imgcreate-13.4.5-2.el7~mos1.x86_64 from mos excluded (priority)\n', ' --> python-jsonpointer-1.9-2.el7~mos1.noarch from mos excluded (priority)\n', ' --> python-ply-3.4-4.el7~mos2.noarch from mos excluded (priority)\n', ' --> python-pycparser-2.10-1.el7~mos1.noarch from mos excluded (priority)\n',
...

Reproducibility:
 - almost all systests
Workaround:
 -
Impact:
 - Yum output require additional transformation locally to become readable.

Description of the environment:
 Operation system: CentOS-7
 Versions of components: -
 Reference architecture: -
 Network model: -
 Related projects installed: -
Additional information:
 -

Revision history for this message
Aleksandra Fedorova (bookwar) wrote :

sys_test.log is produced by fuel-qa framework and archived by Jenkins as it is.

QA team, please take a look if formatting can be improved

Changed in fuel:
status: New → Confirmed
status: Confirmed → New
assignee: Fuel CI (fuel-ci) → Fuel QA Team (fuel-qa)
tags: added: area-qa
removed: area-ci
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

@Igor, you can find file "packages.json" after each job, so it is more readable format for packages. And we cannot add into debug output a list with all packages, I guess you should understand "why".

Changed in fuel:
importance: High → Wishlist
status: New → Won't Fix
Changed in fuel:
importance: Wishlist → High
status: Won't Fix → Confirmed
Revision history for this message
Igor Yozhikov (iyozhikov) wrote :

@Nastya
Mentioned by you file absent as job artifact - https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-centos/578/

I really don't understand "why", log produced by yum package manager can't be saved as is.

It is not a wishlist - because package maintainers should be able to troubleshoot failures related to packages, and clear and formatted logs playing here significant role and making debug process more quick and not complex.

Revision history for this message
Alexander Tsamutali (astsmtl) wrote :

Maybe we can write structures using pprint?

Changed in fuel:
importance: High → Medium
Revision history for this message
Igor Yozhikov (iyozhikov) wrote :

Yum package manager doesn't log any not finished successfully transactions and information related to yum fails couldn't be logged to yum.log file.

Since that modifications should be made here(run command with additional logging into file):
https://github.com/openstack/fuel-qa/blob/eaea19b42fcfb84e8936187852f0355ebda88a17/fuelweb_test/helpers/decorators.py#L238
yum update -y -d3
yum update -y -d3 2>>/var/log/yum.error.log

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Igor Yozhikov (iyozhikov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/320934

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/320934
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7059be0d2e7e965d9607be78c35e0b1b550a2681
Submitter: Jenkins
Branch: master

commit 7059be0d2e7e965d9607be78c35e0b1b550a2681
Author: Igor Yozhikov <email address hidden>
Date: Wed May 25 15:09:21 2016 +0300

    Add additional logging for yum failures

     * add yum update failures logging into /var/log/yum-update-error.log
     shotgun will pick up this file and include into diagnostic snapshot
     * add /var/log/yum-update-error.log to snapshot log template

    Change-Id: I3ff35eb78c1b5f85275654a18d1436173aa80ebb
    Closes-Bug: #1585194

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/335428

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/8.0)

Reviewed: https://review.openstack.org/321611
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=1e69474a304a619d64c0def48d7eb74e83690e9d
Submitter: Jenkins
Branch: stable/8.0

commit 1e69474a304a619d64c0def48d7eb74e83690e9d
Author: Alexey Stepanov <email address hidden>
Date: Thu May 26 14:51:23 2016 +0300

    Upgrade rework

    1. Rebased on upgrade_ceph_ha_restore
      (master node is 8.0, fuel-octane is already installed)
    2. Extended tests amount to upgrade whole cluster and check every step
      (previously checks was skipped)
    3. Run on remote helper: logging is aligned to the modern ssh manager for easier debugging
      (use str instead of list of binary strings -> human readable error output. Related bug found)

    Related-Bug:1585194
    Change-Id: Ibee784b9cc4be8b2628e7646c51a21910febbb9e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/339012

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/339021

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/339021
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=51ed3b4972ee7b66195744c8b11110079b8828f2
Submitter: Jenkins
Branch: master

commit 51ed3b4972ee7b66195744c8b11110079b8828f2
Author: Alexey Stepanov <email address hidden>
Date: Thu May 26 14:51:23 2016 +0300

    Upgrade rework

    1. Rebased on upgrade_ceph_ha_restore
      (master node is 8.0, fuel-octane is already installed)
    2. Extended tests amount to upgrade whole cluster and check every step
      (previously checks was skipped)
    3. Run on remote helper: logging is aligned to the modern ssh manager for easier debugging
      (use str instead of list of binary strings -> human readable error output. Related bug found)

    Related-Bug:1585194
    Change-Id: Ibee784b9cc4be8b2628e7646c51a21910febbb9e
    (cherry picked from commit 1e69474)
    (cherry picked from commit c404a62)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/339012
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c404a62157d36fb996c950c81d46c7d2a405474b
Submitter: Jenkins
Branch: stable/mitaka

commit c404a62157d36fb996c950c81d46c7d2a405474b
Author: Alexey Stepanov <email address hidden>
Date: Thu May 26 14:51:23 2016 +0300

    Upgrade rework

    1. Rebased on upgrade_ceph_ha_restore
      (master node is 8.0, fuel-octane is already installed)
    2. Extended tests amount to upgrade whole cluster and check every step
      (previously checks was skipped)
    3. Run on remote helper: logging is aligned to the modern ssh manager for easier debugging
      (use str instead of list of binary strings -> human readable error output. Related bug found)

    Related-Bug:1585194
    Change-Id: Ibee784b9cc4be8b2628e7646c51a21910febbb9e
    (cherry picked from commit 1e69474)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/343747

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/344165

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/8.0)

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/344200

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/345966

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/8.0)

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/345970

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/335428
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=41c2818c0d202f58dbee836be2b3c9d856e96605
Submitter: Jenkins
Branch: master

commit 41c2818c0d202f58dbee836be2b3c9d856e96605
Author: Alexander Kurenyshev <email address hidden>
Date: Wed Jun 29 13:50:28 2016 +0300

    Patch adds yum logging to the updates code

    Previously fix was added to the decorator.
    This one will cover updates section

    Change-Id: I79495b19d341023a5f5067379d3ead2a5b8d3075
    Related-Bug:1585194

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/345966
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=1dafe40068d7ba9d96899b79b826ba22a4345c86
Submitter: Jenkins
Branch: stable/mitaka

commit 1dafe40068d7ba9d96899b79b826ba22a4345c86
Author: Alexander Kurenyshev <email address hidden>
Date: Wed Jun 29 13:50:28 2016 +0300

    Patch adds yum logging to the updates code

    Previously fix was added to the decorator.
    This one will cover updates section

    Change-Id: I79495b19d341023a5f5067379d3ead2a5b8d3075
    Related-Bug:1585194
    (cherry picked from commit 8cd2007a1904e969976cce6d8abca84386d127f5)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/8.0)

Reviewed: https://review.openstack.org/345970
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7b4e22c9713b9bdaa5495607038e34f8082e3338
Submitter: Jenkins
Branch: stable/8.0

commit 7b4e22c9713b9bdaa5495607038e34f8082e3338
Author: Alexander Kurenyshev <email address hidden>
Date: Wed Jun 29 13:50:28 2016 +0300

    Patch adds yum logging to the updates code

    Previously fix was added to the decorator.
    This one will cover updates section

    Change-Id: I79495b19d341023a5f5067379d3ead2a5b8d3075
    Related-Bug:1585194
    (cherry picked from commit 8cd2007a1904e969976cce6d8abca84386d127f5)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (master)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/343747
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (stable/mitaka)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/344165
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (stable/8.0)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/344200
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.