kdump is not captured in remote host when kdump over ssh is configured
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
The Ubuntu-power-systems project |
Fix Released
|
Low
|
Unassigned | ||
makedumpfile (Ubuntu) |
Fix Released
|
Medium
|
Guilherme G. Piccoli | ||
Xenial |
Won't Fix
|
Medium
|
Thadeu Lima de Souza Cascardo | ||
Bionic |
Fix Released
|
Medium
|
Thadeu Lima de Souza Cascardo | ||
Cosmic |
Won't Fix
|
Medium
|
Thadeu Lima de Souza Cascardo | ||
Disco |
Fix Released
|
Medium
|
Thadeu Lima de Souza Cascardo | ||
Eoan |
Fix Released
|
Medium
|
Guilherme G. Piccoli |
Bug Description
[Impact]
* Kdump over network (like NFS mount or SSH dump) relies on network-online target from systemd. Even so, there are some NICs that report "Link Up" state but aren't ready to transmit packets. This is a generally bad behavior that is credited probably to NIC firmware delays, usually not fixable from drivers. Some adapters known to act like this are bnx2x, tg3 and ixgbe.
* Kdump is a mechanism that may be a last resort to debug complex/hard to reproduce issues, so it's interesting to increase its reliability / resilience. We then propose here a solution/quirk to this issue on network dump by adding a retry/delay mechanism; if it's a network dump, kdump will retry some times and sleep between the attempts in order to exclude the case of NICs that aren't ready yet but will soon be able to transmit packets.
* Although first reported by IBM in PowerPC arch, the scope for this issue is the NIC, and it was later reported in x86 arch too.
[Test case]
Usually it's difficult to naturally reproduce this issue in a deterministic way, but we have an artificial test case on comment #24 of this LP.
Also, we have a report from this bug in which the user managed to reproduce the problem consistently - it's fixed after testing our solution.
[Regression potential]
There's not a clear regression potential here since it's just a retry/delay mechanism. Some potential problems may come from bad coding in the script.
The delay between attempts is only 3 sec per iteration, so it shouldn't block the kdump progress for a high amount of time at once.
[Other information]
Salsa Debian commit:
https:/
Changed in makedumpfile (Ubuntu): | |
assignee: | Taco Screen team (taco-screen-team) → Nish Aravamudan (nacc) |
importance: | Undecided → High |
tags: | added: ubuntu-17.04 |
Changed in ubuntu-power-systems: | |
importance: | Undecided → High |
tags: | added: triage-a |
Changed in makedumpfile (Ubuntu): | |
assignee: | Nish Aravamudan (nacc) → nobody |
Changed in makedumpfile (Ubuntu): | |
assignee: | nobody → Canonical Kernel Team (canonical-kernel-team) |
tags: | added: kernel-da-key |
Changed in ubuntu-power-systems: | |
assignee: | nobody → Canonical Kernel Team (canonical-kernel-team) |
tags: |
added: triage-r removed: triage-a |
tags: | added: ppc64el-kdump |
Changed in ubuntu-power-systems: | |
status: | New → Triaged |
summary: |
- Ubuntu 17.04: dump is not captured in remote host when kdump over ssh is - configured on firestone. + dump is not captured in remote host when kdump over ssh is configured on + firestone. |
tags: | removed: ubuntu-17.04 |
summary: |
- dump is not captured in remote host when kdump over ssh is configured on - firestone. + [18.10]dump is not captured in remote host when kdump over ssh is + configured on firestone. |
summary: |
- [18.10]dump is not captured in remote host when kdump over ssh is - configured on firestone. + [Feat req18.10]dump is not captured in remote host when kdump over ssh + is configured on firestone. |
Changed in ubuntu-power-systems: | |
importance: | High → Low |
Changed in makedumpfile (Ubuntu): | |
importance: | High → Low |
summary: |
- [Feat req18.10]dump is not captured in remote host when kdump over ssh - is configured on firestone. + [Feat 18.10]dump is not captured in remote host when kdump over ssh is + configured on firestone. |
tags: | removed: bugnameltc-152306 kernel-da-key ppc64el-kdump severity-high triage-g |
tags: | added: kernel-da-key ppc64el-kdump triage-g |
summary: |
- [Feat 18.10]dump is not captured in remote host when kdump over ssh is + [FEAT 18.10] dump is not captured in remote host when kdump over ssh is configured on firestone. |
Changed in makedumpfile (Ubuntu): | |
assignee: | Canonical Kernel Team (canonical-kernel-team) → nobody |
Changed in ubuntu-power-systems: | |
assignee: | Canonical Kernel Team (canonical-kernel-team) → nobody |
Changed in makedumpfile (Ubuntu): | |
status: | New → Incomplete |
summary: |
- [FEAT 18.10] dump is not captured in remote host when kdump over ssh is - configured on firestone. + kdump is not captured in remote host when kdump over ssh is configured |
no longer affects: | makedumpfile (Ubuntu) |
Changed in makedumpfile (Ubuntu): | |
status: | New → Confirmed |
importance: | Undecided → Medium |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in makedumpfile (Ubuntu Xenial): | |
status: | New → Confirmed |
Changed in makedumpfile (Ubuntu Bionic): | |
status: | New → Confirmed |
Changed in makedumpfile (Ubuntu Cosmic): | |
status: | New → Confirmed |
Changed in makedumpfile (Ubuntu Disco): | |
status: | New → Confirmed |
Changed in makedumpfile (Ubuntu Xenial): | |
importance: | Undecided → Medium |
Changed in makedumpfile (Ubuntu Bionic): | |
importance: | Undecided → Medium |
Changed in makedumpfile (Ubuntu Cosmic): | |
importance: | Undecided → Medium |
Changed in makedumpfile (Ubuntu Disco): | |
importance: | Undecided → Medium |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in makedumpfile (Ubuntu Cosmic): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in makedumpfile (Ubuntu Bionic): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
Changed in makedumpfile (Ubuntu Xenial): | |
assignee: | nobody → Guilherme G. Piccoli (gpiccoli) |
tags: |
added: sts removed: targetmilestone-inin16043 |
Changed in ubuntu-power-systems: | |
status: | Incomplete → Confirmed |
Changed in ubuntu-power-systems: | |
status: | Confirmed → In Progress |
description: | updated |
tags: | added: sts-sponsor-slashd |
Changed in ubuntu-power-systems: | |
status: | In Progress → Fix Committed |
tags: | removed: sts-sponsor-slashd |
Changed in ubuntu-power-systems: | |
status: | In Progress → Fix Released |
Default Comment by Bridge