dbginfo.sh does not complete on Ubuntu 16.04(.5)

[Frank Heimes]

dbginfo.sh does not complete on Ubuntu 16.04(.5) using s390-tools version 1.34.0-0ubuntu8.7

Execution starts and proceeds until step 4 'sysfs':
$ sudo dbginfo.sh
dbginfo.sh: Debug information script version 1.34.0-build-20180502
Copyright IBM Corp. 2002, 2016

Hardware platform = s390x
Kernel version = 4.15.0 (4.15.0-33-generic)
Runtime environment = LPAR

1 of 10: Collecting command output

2 of 10: Collecting z/VM command output skipped - no z/VM environment

3 of 10: Collecting procfs

4 of 10: Collecting sysfs

'dd' process with 100% CPU can be seen:

# ps aux | grep [d]d\
root 257158 100 0.0 2120 664 pts/0 R+ 06:51 11:18 dd if=/sys/kernel/debug/tracing/trace status=noxfer iflag=nonblock of=/tmp/DBGINFO-2018-10-22-06-49-52-s1lp15-1F8F67//sys/kernel/debug/tracing/trace

Happens on 16.04.5 with kernel 4.4 or HWE kernel 4.15.
But works fine on 18.04 (with kernel 4.15) and s390-tools 2.3.0-0ubuntu3.

----------

The regression risk of this fixed dbginfo.sh script and the updated s390-tools package is very low for the following reasons:

- the script is broken today on xenial, hence it cannot really get that worse...
- the fix was provided by IBM via a newer s390-tools package and was pretested by IBM before it was made available
- all this is limited to the s390x architecture and also to xenial (newer Ubuntu releases use newer packages)
- newer s390-tools packages that already incl. the fix are used in newer Ubuntu server releases and are known to work there

[Frank Heimes (fheimes)]

missing dump2tar in s390-tools 1.34.0
w/o dump2tar dbginfo.sh hangs in case huge amounts of devices are available

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-10-29 09:18 EDT-------
This function will be made available with 1.370. https://www.ibm.com/developerworks/linux/linux390/s390-tools-1.37.0.html and need to be cherry picked for 1.34.0.
- dump2tar
- dbginfo.sh

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-11-28 06:01 EDT-------
@Canonical. any update available, how to proceed here.. Thx

[Frank Heimes (fheimes)]

We will probably pick the missing tool and the updated script from 1.34 to get this issue solved on 16.04.5.

[Dimitri John Ledkov (xnox)]

Hello.

I have prepared this backport, however I am not sure if I did right.

Would you please be able to test the build from a PPA?

s390-tools (1.34.0-0ubuntu8.9) xenial; urgency=medium

  * dump2tar: backport utility
  * dbginfo.sh: backport patch to use dump2tar instead of walking sysfs
  * Addresses dbginfo.sh hanging indefinately whilst walking sysfs. LP:
    #1799208

 -- Dimitri John Ledkov <email address hidden> Wed, 23 Jan 2019 07:36:29 -0500

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3608

sudo add-apt-repository ppa:ci-train-ppa-service/3608
sudo apt-get update
sudo apt-get upgrade

[Frank Heimes (fheimes)]

I can confirm that the patched package works fine!

I just tested two packages:
1) 1.34.0-0ubuntu8.8 (neg. test)
still hangs
2) 1.34.0-0ubuntu8.9 (PPA)
now works like expected !

$ apt-cache policy s390-tools
s390-tools:
  Installed: 1.34.0-0ubuntu8.9
  Candidate: 1.34.0-0ubuntu8.9
  Version table:
 *** 1.34.0-0ubuntu8.9 500
        500 http://ppa.launchpad.net/ci-train-ppa-service/3608/ubuntu xenial/main s390x Packages
        100 /var/lib/dpkg/status
     1.34.0-0ubuntu8.8 500
        500 http://us.ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x Packages
     1.34.0-0ubuntu8.7 500
        500 http://ports.ubuntu.com/ubuntu-ports xenial-security/main s390x Packages
     1.34.0-0ubuntu8 500
        500 http://us.ports.ubuntu.com/ubuntu-ports xenial/main s390x Packages
$ sudo dbginfo.sh
dbginfo.sh: Debug information script version 1.34.0-build-20190123
Copyright IBM Corp. 2002, 2016

Hardware platform = s390x
Kernel version = 4.15.0 (4.15.0-33-generic)
Runtime environment = LPAR

1 of 10: Collecting command output

2 of 10: Collecting z/VM command output skipped - no z/VM environment

3 of 10: Collecting procfs

4 of 10: Collecting sysfs

5 of 10: Collecting log files

6 of 10: Collecting config files

7 of 10: Collecting osa oat output

8 of 10: Collecting OpenVSwitch output skipped

9 of 10: Collecting domain xml files

10 of 10: Postprocessing

Finalizing: Creating archive with collected data

Collected data was saved to:
 >> /tmp/DBGINFO-2019-01-31-08-27-43-s1lp15-1F8F67.tgz <<

Review the collected data before sending to your service organization.

$

[Dimitri John Ledkov (xnox)]

In unapproved queue

[Frank Heimes (fheimes)]

Many thx!

[Brian Murray (brian-murray)]

Hello Frank, or anyone else affected,

Accepted s390-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.34.0-0ubuntu8.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

The test case seems pretty straightforwards so I've approved the SRU but the bug description could still use information about regression potential.

[Frank Heimes (fheimes)]

Successfully verified on xenial using proposed package - adjusted the tags accordingly.

Added a section regarding regression risk to the bug description.

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 1.34.0-0ubuntu8.9

---------------
s390-tools (1.34.0-0ubuntu8.9) xenial; urgency=medium

  * dump2tar: backport utility
  * dbginfo.sh: backport patch to use dump2tar instead of walking sysfs
  * Addresses dbginfo.sh hanging indefinately whilst walking sysfs. LP:
    #1799208

 -- Dimitri John Ledkov <email address hidden> Wed, 23 Jan 2019 07:36:29 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for s390-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-02-13 02:37 EDT-------
IBM bugzilla status-> closed, Fix Released for Xenial