Activity log for bug #1681909

Date Who What changed Old value New value Message
2017-04-11 18:39:57 bugproxy bug added bug
2017-04-11 18:40:00 bugproxy tags architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin1704
2017-04-11 18:40:02 bugproxy attachment added console log https://bugs.launchpad.net/bugs/1681909/+attachment/4860328/+files/kdump_over_ssh_firestone_NV
2017-04-11 18:40:05 bugproxy attachment added console log https://bugs.launchpad.net/bugs/1681909/+attachment/4860329/+files/kdump_over_ssh_firestone_NV
2017-04-11 18:40:22 bugproxy attachment added sosreport https://bugs.launchpad.net/bugs/1681909/+attachment/4860330/+files/sosreport-ltc-firep3-20170308001351.tar.xz
2017-04-11 18:40:25 bugproxy attachment added Console log of successful dump capture after adding a time delay of 'sleep 30' https://bugs.launchpad.net/bugs/1681909/+attachment/4860331/+files/kdump-with-sleep30-ssh.log
2017-04-11 18:40:26 bugproxy ubuntu: assignee Taco Screen team (taco-screen-team)
2017-04-11 18:40:31 bugproxy affects ubuntu makedumpfile (Ubuntu)
2017-04-26 14:15:59 Manoj Iyer bug task added ubuntu-power-systems
2017-05-08 19:40:20 Manoj Iyer makedumpfile (Ubuntu): assignee Taco Screen team (taco-screen-team) Nish Aravamudan (nacc)
2017-05-08 19:40:25 Manoj Iyer makedumpfile (Ubuntu): importance Undecided High
2017-06-01 15:18:05 Manoj Iyer tags architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin1704 architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin1704 ubuntu-17.04
2017-07-10 10:49:42 bugproxy tags architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin1704 ubuntu-17.04 architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin16043 ubuntu-17.04
2017-07-19 16:04:52 Manoj Iyer ubuntu-power-systems: importance Undecided High
2017-07-31 14:38:02 Manoj Iyer tags architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin16043 ubuntu-17.04 architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin16043 triage-a ubuntu-17.04
2017-08-14 17:29:53 David Britton makedumpfile (Ubuntu): assignee Nish Aravamudan (nacc)
2017-08-14 17:45:19 Manoj Iyer makedumpfile (Ubuntu): assignee Canonical Kernel Team (canonical-kernel-team)
2017-08-14 19:04:43 Joseph Salisbury tags architecture-ppc64le bugnameltc-152306 severity-high targetmilestone-inin16043 triage-a ubuntu-17.04 architecture-ppc64le bugnameltc-152306 kernel-da-key severity-high targetmilestone-inin16043 triage-a ubuntu-17.04
2017-08-21 13:49:59 Andrew Cloke ubuntu-power-systems: assignee Canonical Kernel Team (canonical-kernel-team)
2017-09-11 13:51:41 Manoj Iyer tags architecture-ppc64le bugnameltc-152306 kernel-da-key severity-high targetmilestone-inin16043 triage-a ubuntu-17.04 architecture-ppc64le bugnameltc-152306 kernel-da-key severity-high targetmilestone-inin16043 triage-r ubuntu-17.04
2017-11-28 17:10:55 Andrew Cloke tags architecture-ppc64le bugnameltc-152306 kernel-da-key severity-high targetmilestone-inin16043 triage-r ubuntu-17.04 architecture-ppc64le bugnameltc-152306 kernel-da-key ppc64el-kdump severity-high targetmilestone-inin16043 triage-r ubuntu-17.04
2017-12-04 14:36:12 Andrew Cloke ubuntu-power-systems: status New Triaged
2018-02-26 14:51:24 Andrew Cloke ubuntu-power-systems: status Triaged Incomplete
2018-02-26 14:51:43 Andrew Cloke tags architecture-ppc64le bugnameltc-152306 kernel-da-key ppc64el-kdump severity-high targetmilestone-inin16043 triage-r ubuntu-17.04 architecture-ppc64le bugnameltc-152306 kernel-da-key ppc64el-kdump severity-high targetmilestone-inin16043 triage-g ubuntu-17.04
2018-03-05 15:41:34 Andrew Cloke summary Ubuntu 17.04: dump is not captured in remote host when kdump over ssh is configured on firestone. dump is not captured in remote host when kdump over ssh is configured on firestone.
2018-03-05 15:41:37 Frank Heimes tags architecture-ppc64le bugnameltc-152306 kernel-da-key ppc64el-kdump severity-high targetmilestone-inin16043 triage-g ubuntu-17.04 architecture-ppc64le bugnameltc-152306 kernel-da-key ppc64el-kdump severity-high targetmilestone-inin16043 triage-g
2018-03-05 15:55:06 bugproxy attachment added Console log of successful dump capture after adding a time delay of 'sleep 30' https://bugs.launchpad.net/bugs/1681909/+attachment/5070039/+files/kdump-with-sleep30-ssh.log
2018-03-06 14:40:44 bugproxy attachment added Console log - Kdump failure attempt with Ubuntu 16.04 https://bugs.launchpad.net/bugs/1681909/+attachment/5070786/+files/console_kdump_1604.log
2018-03-06 16:31:43 bugproxy attachment added Dmesg - Kdump failed attempt with Ubuntu 16.04 https://bugs.launchpad.net/bugs/1681909/+attachment/5070860/+files/dmesg_kdump_ubuntu1604.txt
2018-03-19 14:08:50 Manoj Iyer summary dump is not captured in remote host when kdump over ssh is configured on firestone. [18.10]dump is not captured in remote host when kdump over ssh is configured on firestone.
2018-03-19 14:09:08 Manoj Iyer summary [18.10]dump is not captured in remote host when kdump over ssh is configured on firestone. [Feat req18.10]dump is not captured in remote host when kdump over ssh is configured on firestone.
2018-03-19 14:28:22 Manoj Iyer ubuntu-power-systems: importance High Low
2018-03-19 14:28:24 Manoj Iyer makedumpfile (Ubuntu): importance High Low
2018-04-05 16:11:42 Frank Heimes summary [Feat req18.10]dump is not captured in remote host when kdump over ssh is configured on firestone. [Feat 18.10]dump is not captured in remote host when kdump over ssh is configured on firestone.
2018-04-05 16:29:23 bugproxy tags architecture-ppc64le bugnameltc-152306 kernel-da-key ppc64el-kdump severity-high targetmilestone-inin16043 triage-g architecture-ppc64le targetmilestone-inin16043
2018-05-14 17:22:50 Frank Heimes tags architecture-ppc64le targetmilestone-inin16043 architecture-ppc64le kernel-da-key ppc64el-kdump targetmilestone-inin16043 triage-g
2018-07-02 13:39:41 Frank Heimes summary [Feat 18.10]dump is not captured in remote host when kdump over ssh is configured on firestone. [FEAT 18.10] dump is not captured in remote host when kdump over ssh is configured on firestone.
2019-05-29 16:46:13 Manoj Iyer makedumpfile (Ubuntu): assignee Canonical Kernel Team (canonical-kernel-team)
2019-05-29 16:46:17 Manoj Iyer ubuntu-power-systems: assignee Canonical Kernel Team (canonical-kernel-team)
2019-06-10 13:52:25 Manoj Iyer makedumpfile (Ubuntu): status New Incomplete
2019-06-12 19:14:48 Guilherme G. Piccoli summary [FEAT 18.10] dump is not captured in remote host when kdump over ssh is configured on firestone. kdump is not captured in remote host when kdump over ssh is configured
2019-06-12 19:14:55 Guilherme G. Piccoli bug task deleted makedumpfile (Ubuntu)
2019-06-12 19:16:07 Guilherme G. Piccoli bug task added makedumpfile (Ubuntu)
2019-06-12 19:16:16 Guilherme G. Piccoli makedumpfile (Ubuntu): status New Confirmed
2019-06-12 19:16:18 Guilherme G. Piccoli makedumpfile (Ubuntu): importance Undecided Medium
2019-06-12 19:16:21 Guilherme G. Piccoli makedumpfile (Ubuntu): assignee Guilherme G. Piccoli (gpiccoli)
2019-06-12 19:16:47 Guilherme G. Piccoli bug added subscriber Murilo Fossa Vicentini
2019-06-12 19:17:27 Guilherme G. Piccoli nominated for series Ubuntu Bionic
2019-06-12 19:17:27 Guilherme G. Piccoli bug task added makedumpfile (Ubuntu Bionic)
2019-06-12 19:17:27 Guilherme G. Piccoli nominated for series Ubuntu Cosmic
2019-06-12 19:17:27 Guilherme G. Piccoli bug task added makedumpfile (Ubuntu Cosmic)
2019-06-12 19:17:27 Guilherme G. Piccoli nominated for series Ubuntu Xenial
2019-06-12 19:17:27 Guilherme G. Piccoli bug task added makedumpfile (Ubuntu Xenial)
2019-06-12 19:17:27 Guilherme G. Piccoli nominated for series Ubuntu Eoan
2019-06-12 19:17:27 Guilherme G. Piccoli bug task added makedumpfile (Ubuntu Eoan)
2019-06-12 19:17:27 Guilherme G. Piccoli nominated for series Ubuntu Disco
2019-06-12 19:17:27 Guilherme G. Piccoli bug task added makedumpfile (Ubuntu Disco)
2019-06-12 19:17:37 Guilherme G. Piccoli makedumpfile (Ubuntu Xenial): status New Confirmed
2019-06-12 19:17:39 Guilherme G. Piccoli makedumpfile (Ubuntu Bionic): status New Confirmed
2019-06-12 19:17:41 Guilherme G. Piccoli makedumpfile (Ubuntu Cosmic): status New Confirmed
2019-06-12 19:17:43 Guilherme G. Piccoli makedumpfile (Ubuntu Disco): status New Confirmed
2019-06-12 19:17:45 Guilherme G. Piccoli makedumpfile (Ubuntu Xenial): importance Undecided Medium
2019-06-12 19:17:46 Guilherme G. Piccoli makedumpfile (Ubuntu Bionic): importance Undecided Medium
2019-06-12 19:17:48 Guilherme G. Piccoli makedumpfile (Ubuntu Cosmic): importance Undecided Medium
2019-06-12 19:17:50 Guilherme G. Piccoli makedumpfile (Ubuntu Disco): importance Undecided Medium
2019-06-12 19:17:51 Guilherme G. Piccoli makedumpfile (Ubuntu Disco): assignee Guilherme G. Piccoli (gpiccoli)
2019-06-12 19:17:52 Guilherme G. Piccoli makedumpfile (Ubuntu Cosmic): assignee Guilherme G. Piccoli (gpiccoli)
2019-06-12 19:17:54 Guilherme G. Piccoli makedumpfile (Ubuntu Bionic): assignee Guilherme G. Piccoli (gpiccoli)
2019-06-12 19:17:56 Guilherme G. Piccoli makedumpfile (Ubuntu Xenial): assignee Guilherme G. Piccoli (gpiccoli)
2019-06-12 19:18:40 Guilherme G. Piccoli tags architecture-ppc64le kernel-da-key ppc64el-kdump targetmilestone-inin16043 triage-g architecture-ppc64le kernel-da-key ppc64el-kdump sts triage-g
2019-06-24 14:18:24 Andrew Cloke ubuntu-power-systems: status Incomplete Confirmed
2019-07-04 20:02:11 Guilherme G. Piccoli attachment added 0001-NOT-UPSTREAM-virtio-net-Add-droppkt-param-to-force-d.patch https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+attachment/5275113/+files/0001-NOT-UPSTREAM-virtio-net-Add-droppkt-param-to-force-d.patch
2019-07-04 20:33:51 Guilherme G. Piccoli description == Comment: #0 - PAVITHRA R. PRAKASH <pavrampu@in.ibm.com> - 2017-03-07 05:00:29 == ---Problem Description--- Ubuntu 17.04: dump is not captured in remote host when kdump over ssh is configured on firestone. ---Steps to Reproduce--- 1. Configure kdump. 2. Check whether kdump is operational using ?# kdump-config show?. 3. Install ?kernel-debuginfo? and ?kernel-debuginfo-common? rpms. 4. Setup password less ssh connection, generate rsa key. # ssh-keygen -t rsa 5. verify id_rsa and id_rsa.pub are created under /root/.ssh/ 6. Edit /etc/default/kdump-tools and add below entries. SSH="ubuntu@9.114.15.239" SSH_KEY=/root/.ssh/id_rsa 7. Propagate RSA key. # kdump-config propagate 8. Restart kdump service. # kdump-config load 9. Trigger Crash using below commands. # echo "1" > /proc/sys/kernel/sysrq # echo "c" > /proc/sysrq-trigger 10. Verify dump is available in remote server in configured path. Machine details =========== $ ipmitool -I lanplus -H 9.47.70.3 -U ADMIN -P admin sol activate $ ssh ubuntu@9.47.70.29 PW: shriya101 Attaching logs == Comment: #1 - PAVITHRA R. PRAKASH <pavrampu@in.ibm.com> - 2017-03-07 05:01:42 == == Comment: #5 - PAVITHRA R. PRAKASH <pavrampu@in.ibm.com> - 2017-03-07 23:19:46 == Hi, Attaching the logs. Network info: root@ltc-firep3:~# hwinfo --network 36: None 00.0: 10700 Loopback [Created at net.126] Unique ID: ZsBS.GQNx7L4uPNA SysFS ID: /class/net/lo Hardware Class: network interface Model: "Loopback network interface" Device File: lo Link detected: yes Config Status: cfg=new, avail=yes, need=no, active=unknown 37: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: 2lHw.ndpeucax6V1 Parent ID: mIXc.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f2 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.2 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f2 HW Address: 98:be:94:03:18:4a Permanent HW Address: 98:be:94:03:18:4a Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #15 (Ethernet controller) 38: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: 7Onn.ndpeucax6V1 Parent ID: sx0U.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f0 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.0 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f0 HW Address: 98:be:94:03:18:48 Permanent HW Address: 98:be:94:03:18:48 Link detected: yes Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #16 (Ethernet controller) 39: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: VwX_.ndpeucax6V1 Parent ID: DUng.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f3 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.3 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f3 HW Address: 98:be:94:03:18:4b Permanent HW Address: 98:be:94:03:18:4b Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #25 (Ethernet controller) 40: None 00.0: 10701 Ethernet [Created at net.126] Unique ID: bZ1s.ndpeucax6V1 Parent ID: J7HY.aXC4wIvegH8 SysFS ID: /class/net/enP33p3s0f1 SysFS Device Link: /devices/pci0021:00/0021:00:00.0/0021:01:00.0/0021:02:01.0/0021:03:00.1 Hardware Class: network interface Model: "Ethernet network interface" Driver: "tg3" Driver Modules: "tg3" Device File: enP33p3s0f1 HW Address: 98:be:94:03:18:49 Permanent HW Address: 98:be:94:03:18:49 Link detected: no Config Status: cfg=new, avail=yes, need=no, active=unknown Attached to: #4 (Ethernet controller) root@ltc-firep3:~# Thanks, Pavithra == Comment: #6 - PAVITHRA R. PRAKASH <pavrampu@in.ibm.com> - 2017-03-07 23:20:47 == == Comment: #7 - PAVITHRA R. PRAKASH <pavrampu@in.ibm.com> - 2017-03-07 23:21:27 == == Comment: #8 - Urvashi Jawere <urjawere@in.ibm.com> - 2017-03-08 02:48:15 == I am able to see some errors in syslog ; auxiliary Mar 7 04:57:44 ltc-firep3 systemd-resolved[3486]: DNSSEC validation failed for question 114.15.239:/home/ubuntu/test IN SOA: failed-auxiliary Mar 7 04:57:44 ltc-firep3 systemd-resolved[3486]: DNSSEC validation failed for question 9.114.15.239:/home/ubuntu/test IN DS: failed-auxiliary Mar 7 04:57:44 ltc-firep3 systemd-resolved[3486]: DNSSEC validation failed for question 9.114.15.239:/home/ubuntu/test IN SOA: failed-auxiliary Mar 7 04:57:44 ltc-firep3 systemd-resolved[3486]: DNSSEC validation failed for question 9.114.15.239:/home/ubuntu/test IN A: failed-auxiliary Mar 7 04:57:44 ltc-firep3 systemd-resolved[3486]: Server 9.12.16.2 does not support DNSSEC, downgrading to non-DNSSEC mode. Mar 7 04:57:44 ltc-firep3 kdump-config: /root/.ssh/id_rsa failed to be sent to ubuntu@9.114.15.239:/home/ubuntu/test Mar 7 04:58:04 ltc-firep3 systemd[1]: Reloading. Mar 7 04:59:15 ltc-firep3 systemd[1]: Reloading. Mar 7 04:59:16 ltc-firep3 kdump-config: propagated ssh key /root/.ssh/id_rsa to server ubuntu@9.114.15.239 . . . Mar 7 05:06:55 ltc-firep3 systemd[1]: Started Accounts Service. Mar 7 05:06:56 ltc-firep3 kdump-tools[3498]: Starting kdump-tools: Modified cmdline:root=UUID=1e76cfd5-988c-46f4-bdc4-39fe1ed01152 ro quiet splash irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service ata_piix.prefer_ms_hyperv=0 elfcorehdr=155136K Mar 7 05:06:57 ltc-firep3 kdump-tools[3498]: * loaded kdump kernel Mar 7 05:06:57 ltc-firep3 kdump-tools: /sbin/kexec -p --command-line="root=UUID=1e76cfd5-988c-46f4-bdc4-39fe1ed01152 ro quiet splash irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz Mar 7 05:06:57 ltc-firep3 kdump-tools: loaded kdump kernel Mar 7 05:06:57 ltc-firep3 systemd[1]: Started Kernel crash dump capture service. Mar 7 05:06:57 ltc-firep3 apport[3584]: ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/linux-image-4.10.0-9-generic-201703060521.crash' Mar 7 05:06:57 ltc-firep3 apport[3584]: ...done. == Comment: #18 - Hari Krishna Bathini <hbathini@in.ibm.com> - 2017-03-28 06:55:20 == Looks like tg3 module was not needed after all. Interesting thing though is even after enP34p1s0f0 is up (ifup) and network.online target is reached, network was not really active. It took about 30 seconds, after reaching network.online target, for the network to be active, even on a normal boot. Adding this wait time in kdump script, before saving dump, ensured that vmcore is captured successful. Attaching the log for the same.. Not sure why enP34p1s0f0 is taking that long to configure/initialize. Even so, this delay should be part of ifup/network-online.target if it is inevitable, so that network is pingable after network-online.target Thanks Hari == Comment: #19 - Hari Krishna Bathini <hbathini@in.ibm.com> - 2017-03-28 07:01:52 == The workaround snippet adding delay in kdump script: --- kdump-config.orig 2017-03-28 03:35:17.753542107 -0500 +++ kdump-config 2017-03-28 06:59:22.887576623 -0500 @@ -761,6 +761,7 @@ KDUMP_DMESGFILE="$KDUMP_STAMPDIR/dmesg.$KDUMP_STAMP" ERROR=0 + sleep 30 ssh -i $KDUMP_SSH_KEY $KDUMP_REMOTE_HOST mkdir -p $KDUMP_STAMPDIR ERROR=$? # If remote connections fails, no need to continue --- Thanks Hari == Comment: #20 - PAVITHRA R. PRAKASH <pavrampu@in.ibm.com> - 2017-03-30 01:33:56 == (In reply to comment #19) > The workaround snippet adding delay in kdump script: > > > --- kdump-config.orig 2017-03-28 03:35:17.753542107 -0500 > +++ kdump-config 2017-03-28 06:59:22.887576623 -0500 > @@ -761,6 +761,7 @@ > KDUMP_DMESGFILE="$KDUMP_STAMPDIR/dmesg.$KDUMP_STAMP" > ERROR=0 > > + sleep 30 > ssh -i $KDUMP_SSH_KEY $KDUMP_REMOTE_HOST mkdir -p $KDUMP_STAMPDIR > ERROR=$? > # If remote connections fails, no need to continue > > --- > > Thanks > Hari With above workaround dump captured successfully in remote host. Thanks, Pavithra == Comment: #22 - Hari Krishna Bathini <hbathini@in.ibm.com> - 2017-04-10 22:14:27 == (In reply to comment #18) > Created attachment 117088 [details] > Console log of successful dump capture after adding a time delay of 'sleep > 30' > > Looks like tg3 module was not needed after all. Interesting thing though is > even after enP34p1s0f0 is up (ifup) and network.online target is reached, > network was not really active. It took about 30 seconds, after reaching > network.online target, for the network to be active, even on a normal boot. > Adding this wait time in kdump script, before saving dump, ensured that > vmcore is captured successful. Attaching the log for the same.. > > Not sure why enP34p1s0f0 is taking that long to configure/initialize. Even > so, > this delay should be part of ifup/network-online.target if it is inevitable, > so that network is pingable after network-online.target Hi Canonical, Since this falls outside the realm of kdump, should we add a NET_WAIT_TIME field in /etc/default/kdump-tools file that defaults to 0 but can be changed when the user sees timing troubles? Thanks Hari [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network-online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once.
2019-07-04 20:34:05 Guilherme G. Piccoli bug added subscriber Thadeu Lima de Souza Cascardo
2019-07-04 20:37:25 Guilherme G. Piccoli attachment added lp1681909_eoan.debdiff https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+attachment/5275117/+files/lp1681909_eoan.debdiff
2019-07-04 20:37:37 Guilherme G. Piccoli makedumpfile (Ubuntu Xenial): status Confirmed In Progress
2019-07-04 20:37:40 Guilherme G. Piccoli makedumpfile (Ubuntu Bionic): status Confirmed In Progress
2019-07-04 20:37:43 Guilherme G. Piccoli makedumpfile (Ubuntu Cosmic): status Confirmed In Progress
2019-07-04 20:37:48 Guilherme G. Piccoli makedumpfile (Ubuntu Disco): status Confirmed In Progress
2019-07-04 20:37:50 Guilherme G. Piccoli makedumpfile (Ubuntu Eoan): status Confirmed In Progress
2019-07-04 20:38:09 Guilherme G. Piccoli makedumpfile (Ubuntu Disco): assignee Guilherme G. Piccoli (gpiccoli) Thadeu Lima de Souza Cascardo (cascardo)
2019-07-04 20:38:17 Guilherme G. Piccoli makedumpfile (Ubuntu Cosmic): assignee Guilherme G. Piccoli (gpiccoli) Thadeu Lima de Souza Cascardo (cascardo)
2019-07-04 20:38:26 Guilherme G. Piccoli makedumpfile (Ubuntu Bionic): assignee Guilherme G. Piccoli (gpiccoli) Thadeu Lima de Souza Cascardo (cascardo)
2019-07-04 20:38:33 Guilherme G. Piccoli makedumpfile (Ubuntu Xenial): assignee Guilherme G. Piccoli (gpiccoli) Thadeu Lima de Souza Cascardo (cascardo)
2019-07-04 20:48:53 Frank Heimes ubuntu-power-systems: status Confirmed In Progress
2019-07-05 00:20:50 Ubuntu Foundations Team Bug Bot tags architecture-ppc64le kernel-da-key ppc64el-kdump sts triage-g architecture-ppc64le kernel-da-key patch ppc64el-kdump sts triage-g
2019-07-05 00:20:56 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Sponsors Team
2019-07-08 02:59:58 Adolfo Jayme bug added subscriber Adolfo Jayme
2019-07-20 19:19:46 Thadeu Lima de Souza Cascardo attachment added SRU for disco https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+attachment/5278173/+files/makedumpfile_1.6.5-1ubuntu1.1_disco.debdiff
2019-07-20 19:20:06 Thadeu Lima de Souza Cascardo attachment added SRU for bionic https://bugs.launchpad.net/ubuntu-power-systems/+bug/1681909/+attachment/5278174/+files/makedumpfile_1.6.5-1ubuntu1~18.04.2_bionic.debdiff
2019-07-22 13:21:54 Guilherme G. Piccoli bug added subscriber STS Sponsors
2019-07-22 16:32:21 Eric Desrochers makedumpfile (Ubuntu Cosmic): status In Progress Won't Fix
2019-07-23 13:47:05 Eric Desrochers description [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network-online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Impact] * Kdump over network (like NFS mount or SSH dump) relies on network-online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe. * Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets. * Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too. [Test case] Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP. Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution. [Regression potential] There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script. The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once. [Other information] Salsa Debian commit: https://salsa.debian.org/debian/makedumpfile/commit/d63ba95337988be1eac8c8c76d90825ff5c6d17f
2019-07-23 14:00:37 Eric Desrochers makedumpfile (Ubuntu Eoan): status In Progress Fix Committed
2019-07-23 14:15:55 Guilherme G. Piccoli makedumpfile (Ubuntu Xenial): status In Progress Won't Fix
2019-07-23 19:49:25 Eric Desrochers bug added subscriber Eric Desrochers
2019-08-16 15:02:44 Eric Desrochers tags architecture-ppc64le kernel-da-key patch ppc64el-kdump sts triage-g architecture-ppc64le kernel-da-key patch ppc64el-kdump sts sts-sponsor-slashd triage-g
2019-08-19 19:58:44 Guilherme G. Piccoli branch linked lp:~gpiccoli/britney/hints-ubuntu
2019-08-20 16:24:14 Launchpad Janitor makedumpfile (Ubuntu Eoan): status Fix Committed Fix Released
2019-08-27 14:33:34 Eric Desrochers removed subscriber Ubuntu Sponsors Team
2019-08-28 13:42:53 Andy Whitcroft makedumpfile (Ubuntu Disco): status In Progress Fix Committed
2019-08-28 13:42:56 Andy Whitcroft bug added subscriber Ubuntu Stable Release Updates Team
2019-08-28 13:42:58 Andy Whitcroft bug added subscriber SRU Verification
2019-08-28 13:43:05 Andy Whitcroft tags architecture-ppc64le kernel-da-key patch ppc64el-kdump sts sts-sponsor-slashd triage-g architecture-ppc64le kernel-da-key patch ppc64el-kdump sts sts-sponsor-slashd triage-g verification-needed verification-needed-disco
2019-08-28 13:50:10 Andy Whitcroft makedumpfile (Ubuntu Bionic): status In Progress Fix Committed
2019-08-28 13:50:17 Andy Whitcroft tags architecture-ppc64le kernel-da-key patch ppc64el-kdump sts sts-sponsor-slashd triage-g verification-needed verification-needed-disco architecture-ppc64le kernel-da-key patch ppc64el-kdump sts sts-sponsor-slashd triage-g verification-needed verification-needed-bionic verification-needed-disco
2019-08-28 14:07:21 Andrew Cloke ubuntu-power-systems: status In Progress Fix Committed
2019-08-29 12:01:28 Eric Desrochers tags architecture-ppc64le kernel-da-key patch ppc64el-kdump sts sts-sponsor-slashd triage-g verification-needed verification-needed-bionic verification-needed-disco architecture-ppc64le kernel-da-key patch ppc64el-kdump sts triage-g verification-needed verification-needed-bionic verification-needed-disco
2019-08-29 12:01:48 Eric Desrochers removed subscriber STS Sponsors
2019-08-30 18:54:08 Guilherme G. Piccoli tags architecture-ppc64le kernel-da-key patch ppc64el-kdump sts triage-g verification-needed verification-needed-bionic verification-needed-disco architecture-ppc64le kernel-da-key patch ppc64el-kdump sts triage-g verification-done verification-done-bionic verification-done-disco
2019-09-30 14:02:44 Andrew Cloke ubuntu-power-systems: status Fix Committed In Progress
2019-09-30 14:02:47 Andrew Cloke makedumpfile (Ubuntu Bionic): status Fix Committed In Progress
2019-09-30 14:02:50 Andrew Cloke makedumpfile (Ubuntu Disco): status Fix Committed In Progress