Remove "ata_piix.prefer_ms_hyperv=0" parameter

[Heather Lemon]

[Impact]

Azure VM instances hit I/O error on boot causing kernel crash

[Test Plan]

Create Ubuntu Marketplace VM on Azure

```
ssh -i .ssh/id_rsa ubuntu@ipaddr<azureVM>
```

Install crash dump utilities (from guide: https://ubuntu.com/server/docs/kernel-crash-dump)

apt-get install kdump-tools

Say (y) to all questions during install

kdump-config show
*shows the vm is not yet ready to kdump

root@bionic3: kdump-config show
 * no crashkernel= parameter in the kernel cmdline
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
   /var/lib/kdump/vmlinuz
kdump initrd:
   /var/lib/kdump/initrd.img
current state: Not ready to kdump

kexec command:
  no kexec command recorded

Reboot the VM

sudo su

As root on the VM after reboot:

kdump-config show

kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x32000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1086-azure
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

#verify kdump is on

cat /proc/cmdline
... crashkernel=512M-:192M

dmesg | grep -i crash
[ 0.071660] kexec: Reserving the low 1M of memory for crashkernel
[ 0.269823] Reserving 192MB of memory at 640MB for crashkernel (System RAM: 4095MB)

cat /proc/sys/kernel/sysrq
# make sure this value is greater than 0
#set it to 1
sudo sysctl -w kernel.sysrq=1

the directory of /var/crash should have no crashes yet as well.

Outcome with "ata_piix.prefer_ms_hyperv=0" in kexec command:
# perform crash
sudo su
echo c > /proc/sysrq-trigger

After a couple of minuties
Open new terminal and try to ssh to azure VM,
It does not succeed

Force a reboot of the VM through the portal or serial console
#kdump doesnt work and hangs indefinitely
#force reboot VM from Azure console

# verify package version of kdump-tool
# verify parameter is not listed in the kdump-config show output
kdump-config unload
kdump-config load
kdump-config show

kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x32000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1086-azure
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

#trigger another crash
echo c > /proc/sysrq-trigger

# open new terminal and ssh back into vm

cd /var/crash

#verify a dump was created
linux-image-5.4.0-1086-azure-202208041658.crash

[Where Problems Could Occur]

This change modifies the debian/rules.
The package could fail to build properly if mistyped.

[Other]
Back-porting a fix from upstream to remove "ata_piix.prefer_ms_hyperv=0" parameter.

target series - Bionic, Focal, Jammy

upstream patch

https://salsa.debian.org/debian/kdump-tools/-/commit/b1bac9396ddbbce3817c34be3161630698e4a503

*Note: There are two source packages needed changes, kdump-tools for Impish -> Kinetic and makedumpfile for series Focal -> Bionic

[Heather Lemon (hypothetical-lemon)]

Jammy debdiff *kdump-tools

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1978333_jammy.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Heather Lemon (hypothetical-lemon)]

It looks like Kinetic has already been resolved

[Heather Lemon (hypothetical-lemon)]

impish debdiff *kdump-tools

[Heather Lemon (hypothetical-lemon)]

debdiff makedumpfile focal

[Heather Lemon (hypothetical-lemon)]

adds bionic makedumpfile patch

[Heather Lemon (hypothetical-lemon)]

For I/J/K in makedumpfile, it seems the default rules used does not have the parameter
ata_piix.prefer_ms_hyperv=0
There is also a changelog entry about removing the parameter:
* ppc64el: Remove ata_piix.prefer_ms_hyperv=0 from crashkernel cmdline

(link to changelog)
https://salsa.debian.org/search?group_id=2&project_id=57654&repository_ref=master&scope=blobs&search=883899

I will mark these as invalid

[Heather Lemon (hypothetical-lemon)]

FYI to build the debian/rules I executed in makedumpfile (I/J/K)

debuild -us -uc -b -rfakeroot

[Steve Langasek (vorlon)]

This SRU test case appears insufficient to me; you are testing in a lxd container software that only has any effect when the kernel crashes. The test case should demonstrate the buggy behavior and that the behavior is fixed by the SRU; a string grepped from kdump-config output is NOT proof of a bug or a fix.

[Heather Lemon (hypothetical-lemon)]

Hi Steve,

The test plan of the description has been updated.

I'm currently testing the Azure bionic vm and confirmed the bug.

The Impish series is EOL and I will remove the series tag.

Thank You,
Heather

[Heather Lemon (hypothetical-lemon)]

EOL

[Heather Lemon (hypothetical-lemon)]

I have followed the steps outlined in the Test Plan for testing the bionic series with the fix from my ppa https://launchpad.net/~hypothetical-lemon/+archive/ubuntu/lp-1978333?field.series_filter=

verified the bug exists in Bionic and has been resolved with the removal of the parameter a crash dump was created and the vm was rebooted without manual intervention.

[Steve Langasek (vorlon)]

Hello Heather, or anyone else affected,

Accepted makedumpfile into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.7-1ubuntu2.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Heather, or anyone else affected,

Accepted makedumpfile into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/makedumpfile/1:1.6.5-1ubuntu1~18.04.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted makedumpfile (1:1.6.5-1ubuntu1~18.04.7) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

makedumpfile/1:1.6.5-1ubuntu1~18.04.7 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#makedumpfile

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Heitor Alves de Siqueira (halves)]

Failed autopkgtest result for i386 seemed to be due to a flaky test, not an
actual regression. Test passed after another run, and no further autopkgtest
regressions have been reported.

[Heather Lemon (hypothetical-lemon)]

I've been testing the Focal/Jammy instances and no crash dump is being generated. I'm still investigating why this is happening.

[Heather Lemon (hypothetical-lemon)]

F/J kernel versions Linux focal-azure 5.15.0-1014-azure
Bionic version - 5.4.0-1086-azure

[Heather Lemon (hypothetical-lemon)]

Heitor advised that upping the kernel crashdump memory from 192MB to 512MB resolved Focal/Jammy not being able to report a crash dump. Thanks Heitor!

[Heather Lemon (hypothetical-lemon)]

# FOCAL - Proposed testing
----------

# enable proposed repo
cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

sudo apt-get install kdump-tools

# ensure propose is selected
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 kdump-tools amd64 1:1.6.7-1ubuntu2.4

Version tested
# dpkg -l | grep kdump
1:1.6.7-1ubuntu2.4

# say yes to all questions

# edit kernel crash params due to this LP[1]
vim /etc/default/grub.d/kdump-tools.cfg

change 192MB to 512MB

save & exit

update-grub

# reboot VM
sudo reboot

# kdump-config show has no ata param listed by default
ubuntu@focal-proposed:~$ sudo su
root@focal-proposed:/home/ubuntu# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x32000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.15.0-1014-azure
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.15.0-1014-azure
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.15.0-1014-azure root=PARTUUID=341f0b90-b764-4a16-a222-6380a3a474f1 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 panic=-1 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

# show upped memory
cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.0-1014-azure root=PARTUUID=341f0b90-b764-4a16-a222-6380a3a474f1 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 crashkernel=512M-:512M panic=-1

# make sure this value is greater than 0
#set it to 1
sudo sysctl -w kernel.sysrq=1

# perform crash
echo c > /proc/sysrq-trigger

kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x1e000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.15.0-1014-azure
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.15.0-1014-azure
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.15.0-1014-azure root=PARTUUID=341f0b90-b764-4a16-a222-6380a3a474f1 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 panic=-1 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

# crash dump file in /var/crash
-rw-r----- 1 root root 18772 Aug 9 18:05 linux-image-5.15.0-1014-azure-202208091804.crash

[1] https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/1918085

[Heather Lemon (hypothetical-lemon)]

# BIONIC - Proposed testing
----------

# enable proposed repo
cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

sudo apt-get install kdump-tools

# ensure propose is selected
Get:8 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 kdump-tools amd64 1:1.6.5-1ubuntu1~18.04.7 [23.6 kB]

Version tested
# dpkg -l | grep kdump
1:1.6.5-1ubuntu1~18.04.7

# say yes to all questions

# edit kernel crash params due to this LP[1]
vim /etc/default/grub.d/kdump-tools.cfg

change 192MB to 512MB

save & exit

update-grub

# reboot VM
sudo reboot

root@bionic-proposed:/home/ubuntu# ls
root@bionic-proposed:/home/ubuntu# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0xd000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1086-azure
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 earlyprintk=ttyS0 crashkernel=512M-:512M

sudo sysctl -w kernel.sysrq=1

# re-shell into into vm
cd /var/crash
rw-r----- 1 root root 17669 Aug 10 16:49 linux-image-5.4.0-1086-azure-202208101649.crash

[Heather Lemon (hypothetical-lemon)]

Is there any updates on the package making into jammy-proposed? Thanks.

[Steve Langasek (vorlon)]

Hello Heather, or anyone else affected,

Accepted kdump-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/kdump-tools/1:1.6.10ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Heather Lemon (hypothetical-lemon)]

# JAMMY - proposed testing
------------

# enable proposed repo
cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

sudo apt-get install kdump-tools

Version tested
dpkg -l | grep kdump
1:1.6.10ubuntu2

# edit kernel crash params due to this LP[1]
vim /etc/default/grub.d/kdump-tools.cfg

change 192MB to 512MB

save & exit

update-grub

# reboot VM
sudo reboot

kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.15.0-1014-azure
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.15.0-1014-azure
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.15.0-1014-azure root=PARTUUID=fbfc827a-3738-445e-9fae-267258b78c1e ro console=tty1 console=ttyS0 earlyprintk=ttyS0 panic=-1 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.0-1014-azure root=PARTUUID=fbfc827a-3738-445e-9fae-267258b78c1e ro console=tty1 console=ttyS0 earlyprintk=ttyS0 crashkernel=512M-:512M panic=-1

# set priority to 1
sudo sysctl -w kernel.sysrq=1

# perform crash
echo c > /proc/sysrq-trigger

cd /var/crash

-rw-r----- 1 root root 18552 Aug 22 13:32 linux-image-5.15.0-1014-azure-202208221332.crash

[Brian Murray (brian-murray)]

kdump-tools (1:1.7) unstable; urgency=medium

  [ Guilherme G. Piccoli ]
  * kdump-tools.service: Start kdump service a bit earlier in boot.

  [ Kellen Renshaw ]
  * rules: Remove "ata_piix.prefer_ms_hyperv=0" (Closes: #1011960).

 -- dann frazier <email address hidden> Thu, 09 Jun 2022 15:29:58 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package kdump-tools - 1:1.6.10ubuntu2

---------------
kdump-tools (1:1.6.10ubuntu2) jammy; urgency=low

  * debian/rules: Remove "ata_piix.prefer_ms_hyperv=0" (LP: #1978333)

 -- Heather Lemon <email address hidden> Fri, 10 Jun 2022 16:26:37 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for kdump-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package makedumpfile - 1:1.6.7-1ubuntu2.4

---------------
makedumpfile (1:1.6.7-1ubuntu2.4) focal; urgency=low

  * debian/rules: Remove "ata_piix.prefer_ms_hyperv=0" (LP: #1978333)

 -- Heather Lemon <email address hidden> Mon, 13 Jun 2022 13:59:29 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package makedumpfile - 1:1.6.5-1ubuntu1~18.04.7

---------------
makedumpfile (1:1.6.5-1ubuntu1~18.04.7) bionic; urgency=medium

  * debian/rules: Remove "ata_piix.prefer_ms_hyperv=0" (LP: #1978333)

 -- Heather Lemon <email address hidden> Mon, 13 Jun 2022 15:19:39 +0000