[UBUNTU 20.04] kernel: unable to read partitions on virtio-block dasd (kvm)

[bugproxy]

SRU Justification:
==================

[Impact]

* The kernel is unable to read partitions on virtio-block DASD (on KVM).
  That's a severe situation, since it prevents Ubuntu from starting, if installed on a DASD.
  This issue can either occur after a fresh installation or after an upgrade.

* The virtio specification virtio-v1.1-cs01 states: "Transitional devices
  MUST detect Legacy drivers by detecting that VIRTIO_F_VERSION_1 has not
  been acknowledged by the driver."
  And this is what QEMU as of 6.1 has done relying solely on
  VIRTIO_F_VERSION_1 for detecting that.

* But the specification also says: "... the driver MAY read (but MUST
  NOT write) the device-specific configuration fields to check that it can
  support the device ..." before setting FEATURES_OK.

* In this case, any transitional device relying solely on VIRTIO_F_VERSION_1
  for detecting legacy drivers will return data in legacy format.
  In particular, this implies that it's in big endian format for
  big endian guests. This naturally confuses the driver that expects
  little endian in the modern mode.

* VIRTIO_F_VERSION_1 can only be relied on after the feature negotiation is done.

* 'verify' is called before virtio_finalize_features(), so a transitional
  s390 virtio device still serves native endian (i.e. big endian) config space,
  while the driver knows that it is going to accept VERSION_1,
  so when reading the config space, it assumes it got little endian, and byteswaps.

* For QEMU, we can work around the issue by writing out the feature bits with
  VIRTIO_F_VERSION_1 bit set. We (ab)use the finalize_features config op for
  this. This isn't enough to address all vhost devices since these do not get
  the features until FEATURES_OK, however it looks like the affected devices
  actually never handled the endianness for legacy mode correctly, so at least
  that's not a regression.

[Fix]

* 2f9a174f918e29608564c7a4e8329893ab604fb4 2f9a174f918e "virtio: write back F_VERSION_1 before validate"

[Test Case]

* Setup an IBM Z or LinuxONE LPAR with Ubuntu Server 20.04 as KVM host.

* This Ubuntu KVM host can either be installed on FCP or DASD storage,
  but at least one DASD disk need to be reserved for a KVM guest.

* Now hand over the reserved DASD disk (low-level formatted using dasdfmt
  and partitioned using fdasd) using 'virtio-block' to a KVM virtual machine
  (e.g. using a virsh VM config).

* Try to install an Ubuntu KVM virtual machine using this DASD disk,
  that includes the check and read of the partition table.

[Where problems could occur]

* First of all requested commit contains one additional if statement;
  and is due tothat relatively traceable.

* But the change is in /drivers/virtio/virtio.c, means in common code!

* This issue obviously affects big endian systems only.

* But if done wrong, it may effect in worst-case little endian systems, too!

* But the if statement explicitly checks for '!virtio_legacy_is_little_endian()'.

* Only virtio net and virtio blk devices seem to be affected.

* And the commit/solutions was in-depth discussed upstream here:
  https://<email address hidden>/t/#u

[Other]

* Patches are upstream accepted with since 5.15-rc6
  and tagged for upstream stable #v4.11.
  Hence jammy is not affected.

* Request was to add the patches to focal / 20.04,
  but to avoid potential regressions on upgrades,
  the patches need to be added to impish and hirsute, too.

* Fortunately cherry-picking the commit works cleanly
  from all the affected Ubuntu releases.

__________

Description: kernel: unable to read partitions on virtio-block dasd (kvm)
Symptom: unable to read partitions on virtio-block dasd (kvm)
Problem: verify is called before virtio_finalize_features() , so a
              transitional s390 virtio device still serves native endian
              (i.e. big endian) config space, while the driver knows that it
              is going to accept VERSION_1, so when reading the config space,
              it assumes it got little endian, and byteswaps.
Solution: For QEMU, we can work around the issue by writing out the
              feature bits with VIRTIO_F_VERSION_1 bit set. We (ab)use the
              finalize_features config op for this. This isn't enough to
              address all vhost devices since these do not get the features
              until FEATURES_OK, however it looks like the affected devices
              actually never handled the endianness for legacy mode correctly,
              so at least that's not a regression.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-11-08 06:34 EDT-------
[Impact]
* unable to read partitions on virtio-block dasd (Regression)

* the problem is that verify is called before virtio_finalize_features() , so a
transitional s390 virtio device still serves native endian
(i.e. big endian) config space, while the driver knows that it
is going to accept VERSION_1, so when reading the config space,
it assumes it got little endian, and byteswaps.

* as a solution for QEMU, we can work around the issue by writing out the
feature bits with VIRTIO_F_VERSION_1 bit set. We (ab)use the
finalize_features config op for this. This isn't enough to
address all vhost devices since these do not get the features
until FEATURES_OK, however it looks like the affected devices
actually never handled the endianness for legacy mode correctly,
so at least that's not a regression.

[Test Plan]
Short: try to read/write virtio-dasd partition table in a kvm guest
Long:
- get access to a dasd device (for this test it is best to pick a dasd that has already been formatted and partitioned in the past,
but can format and partition dasd new if needed).
- configure guest with the full dasd block device as a virtio block device
- boot this guest
- verify whether or not the guest is seeing the pre-created partition on device.

[Fix]

apply kernel upstream commit: 2f9a174f918e (virtio: write back F_VERSION_1 before validate)

[Frank Heimes (fheimes)]

So patch/commit 2f9a174f918e "virtio: write back F_VERSION_1 before validate" got upstream accepted with kernel v5.15-rc6.
With that it needs to be SRUed to impish, hirsute and focal.

I saw that it got tagged for stable updates to #v4.11
Are there plans to tag other stable kernels too, like #5.10.x and/or #5.14.x?

[Frank Heimes (fheimes)]

I mark jammy as Fix Released, since the requested patch got upstream accepted with v5.15-rc6, and kernel 5.15 is the planned target kernel for jammy.

[Frank Heimes (fheimes)]

I'm happy seeing that the commit can be cleanly cherry-picked from all affected releases (impish, hirsute and focal).

[Frank Heimes (fheimes)]

The following PPA contains cross-platform (s390x, ppc64el, arm64 and amd64) test builds for kernels 5.13/impish, 5.11/hirsute and 5.4/focal:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1950144/

[Frank Heimes (fheimes)]

SRU request submitted to the Ubuntu kernel team mailing list for impish, hirsute and focal:
https://lists.ubuntu.com/archives/kernel-team/2021-November/thread.html#125581
Changing status to 'In Progress' for impish, hirsute and focal.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.13.0-23.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.11.0-42.46 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-hirsute' to 'verification-done-hirsute'. If the problem still exists, change the tag 'verification-needed-hirsute' to 'verification-failed-hirsute'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.4.0-92.103 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-12-03 08:37 EDT-------
(In reply to comment #12)
> This bug is awaiting verification that the linux/5.4.0-92.103 kernel in
> -proposed solves the problem. Please test the kernel and update this bug
> with the results. If the problem is solved, change the tag
> 'verification-needed-focal' to 'verification-done-focal'. If the problem
> still exists, change the tag 'verification-needed-focal' to
> 'verification-failed-focal'.
>
> If verification is not done by 5 working days from today, this fix will be
> dropped from the source code, and this bug will be closed.
>
> See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
> enable and use -proposed. Thank you!

The newest kernel I see in proposed is 5.4.0.92.96

root@ubuntu20:~# apt-get install linux-generic/focal-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
linux-generic is already the newest version (5.4.0.92.96).
Selected version '5.4.0.92.96' (Ubuntu:20.04/focal-proposed [s390x]) for 'linux-generic'
The following packages were automatically installed and are no longer required:
linux-headers-5.4.0-18 linux-headers-5.4.0-18-generic
linux-image-5.4.0-18-generic linux-modules-5.4.0-18-generic

Did I mess something up?

[Frank Heimes (fheimes)]

Hi, no you did not messed it up - the version number is causing some confusion from time to time.
The last digit can be ignored (it's just the build number).
If the first 4 digits are the same, you have the right kernel (in other words: content-wise 5.4.0.92.9 is equal to 5.4.0-92.103 - the difference is just that the first one is the source and the second one the binary version).

[bugproxy (bugproxy)]

Hi, no you didn't messed it up - the version number causes some confusion from time to time.
If the first 4 digits are the same, you have the right kernel (in other words: content-wise 5.4.0.92.9 is equal to 5.4.0-92.103 - the difference is just that the first one if the source and the second one the binary version).

------- Comment From <email address hidden> 2021-12-03 11:13 EDT-------
(In reply to comment #17)
> Hi, no you didn't messed it up - the version number causes some confusion
> from time to time.
> The last digit can be ignored (it's just the build number).
> If the first 4 digits are the same, you have the right kernel (in other
> words: content-wise 5.4.0.92.9 is equal to 5.4.0-92.103 - the difference is
> just that the first one if the source and the second one the binary version).

Thank you!

I've verified the fix!

[Frank Heimes (fheimes)]

Ah, many thx - I'm adjusting the tags accordingly ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-92.103

---------------
linux (5.4.0-92.103) focal; urgency=medium

  * focal/linux: 5.4.0-92.103 -proposed tracker (LP: #1952316)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync update-dkms-versions helper
    - debian/dkms-versions -- update from kernel-versions (main/2021.11.29)

  * CVE-2021-4002
    - tlb: mmu_gather: add tlb_flush_*_range APIs
    - hugetlbfs: flush TLBs correctly after huge_pmd_unshare

  * Re-enable DEBUG_INFO_BTF where it was disabled (LP: #1945632)
    - [Config] Enable CONFIG_DEBUG_INFO_BTF on all arches

  * Focal linux-azure: Vm crash on Dv5/Ev5 (LP: #1950462)
    - KVM: VMX: eVMCS: make evmcs_sanitize_exec_ctrls() work again
    - jump_label: Fix usage in module __init

  * Support builtin revoked certificates (LP: #1932029)
    - Revert "UBUNTU: SAUCE: (lockdown) Make get_cert_list() not complain about
      cert lists that aren't present."
    - integrity: Move import of MokListRT certs to a separate routine
    - integrity: Load certs from the EFI MOK config table
    - certs: Add ability to preload revocation certs
    - integrity: Load mokx variables into the blacklist keyring
    - certs: add 'x509_revocation_list' to gitignore
    - SAUCE: Dump stack when X.509 certificates cannot be loaded
    - [Packaging] build canonical-revoked-certs.pem from branch/arch certs
    - [Packaging] Revoke 2012 UEFI signing certificate as built-in
    - [Config] Configure CONFIG_SYSTEM_REVOCATION_KEYS with revoked keys

  * Support importing mokx keys into revocation list from the mok table
    (LP: #1928679)
    - efi: Support for MOK variable config table
    - efi: mokvar-table: fix some issues in new code
    - efi: mokvar: add missing include of asm/early_ioremap.h
    - efi/mokvar: Reserve the table only if it is in boot services data
    - SAUCE: integrity: add informational messages when revoking certs

  * Support importing mokx keys into revocation list from the mok table
    (LP: #1928679) // CVE-2020-26541 when certificates are revoked via
    MokListXRT.
    - SAUCE: integrity: Load mokx certs from the EFI MOK config table

  * Focal update: v5.4.157 upstream stable release (LP: #1951883)
    - ARM: 9133/1: mm: proc-macros: ensure *_tlb_fns are 4B aligned
    - ARM: 9134/1: remove duplicate memcpy() definition
    - ARM: 9139/1: kprobes: fix arch_init_kprobes() prototype
    - ARM: 9141/1: only warn about XIP address when not compile testing
    - ipv6: use siphash in rt6_exception_hash()
    - ipv4: use siphash instead of Jenkins in fnhe_hashfun()
    - usbnet: sanity check for maxpacket
    - usbnet: fix error return code in usbnet_probe()
    - Revert "pinctrl: bcm: ns: support updated DT binding as syscon subnode"
    - ata: sata_mv: Fix the error handling of mv_chip_id()
    - nfc: port100: fix using -ERRNO as command type mask
    - net/tls: Fix flipped sign in tls_err_abort() calls
    - mmc: vub300: fix control-message timeouts
    - mmc: cqhci: clear HALT state after CQE enable
    - mmc: dw_mmc: exynos: fix the finding clock sample value
    - mmc: sdhci: Map more voltage level to SDHCI_POWER_330
    - mmc: sdhci-esdhc-imx: clear the buffe...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.11.0-44.48

---------------
linux (5.11.0-44.48) hirsute; urgency=medium

  * hirsute/linux: 5.11.0-44.48 -proposed tracker (LP: #1954388)

  * Add F81966 watchdog support (LP: #1949063)
    - SAUCE: watchdog: f71808e_wdt: Add F81966 support

linux (5.11.0-42.46) hirsute; urgency=medium

  * hirsute/linux: 5.11.0-42.46 -proposed tracker (LP: #1952278)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync update-dkms-versions helper
    - debian/dkms-versions -- update from kernel-versions (main/2021.11.29)

  * CVE-2021-4002
    - hugetlbfs: flush TLBs correctly after huge_pmd_unshare

  * CVE-2021-43267
    - tipc: fix size validations for the MSG_CRYPTO type

  * Hirsute update: upstream stable patchset 2021-11-24 (LP: #1952136)
    - ext4: check and update i_disksize properly
    - ext4: correct the error path of ext4_write_inline_data_end()
    - ASoC: Intel: sof_sdw: tag SoundWire BEs as non-atomic
    - HID: apple: Fix logical maximum and usage maximum of Magic Keyboard JIS
    - netfilter: ip6_tables: zero-initialize fragment offset
    - HID: wacom: Add new Intuos BT (CTL-4100WL/CTL-6100WL) device IDs
    - ASoC: SOF: loader: release_firmware() on load failure to avoid batching
    - netfilter: nf_nat_masquerade: make async masq_inet6_event handling generic
    - netfilter: nf_nat_masquerade: defer conntrack walk to work queue
    - mac80211: Drop frames from invalid MAC address in ad-hoc mode
    - m68k: Handle arrivals of multiple signals correctly
    - hwmon: (ltc2947) Properly handle errors when looking for the external clock
    - net: prevent user from passing illegal stab size
    - mac80211: check return value of rhashtable_init
    - vboxfs: fix broken legacy mount signature checking
    - net: sun: SUNVNET_COMMON should depend on INET
    - drm/amdgpu: fix gart.bo pin_count leak
    - scsi: ses: Fix unsigned comparison with less than zero
    - scsi: virtio_scsi: Fix spelling mistake "Unsupport" -> "Unsupported"
    - perf/core: fix userpage->time_enabled of inactive events
    - sched: Always inline is_percpu_thread()
    - hwmon: (pmbus/ibm-cffps) max_power_out swap changes
    - ALSA: usb-audio: Unify mixer resume and reset_resume procedure
    - pinctrl: qcom: sc7280: Add PM suspend callbacks
    - io_uring: kill fasync
    - ALSA: usb-audio: Add quirk for VF0770
    - ALSA: pcm: Workaround for a wrong offset in SYNC_PTR compat ioctl
    - ALSA: seq: Fix a potential UAF by wrong private_free call order
    - ALSA: hda/realtek: Enable 4-speaker output for Dell Precision 5560 laptop
    - ALSA: hda - Enable headphone mic on Dell Latitude laptops with ALC3254
    - ALSA: hda/realtek: Complete partial device name to avoid ambiguity
    - ALSA: hda/realtek: Add quirk for Clevo X170KM-G
    - ALSA: hda/realtek - ALC236 headset MIC recording issue
    - ALSA: hda/realtek: Add quirk for TongFang PHxTxX1
    - ALSA: hda/realtek: Fix the mic type detection issue for ASUS G551JW
    - nds32/ftrace: Fix Error: invalid operands (*UND* and *UND* sections) for `^'
    - s390: fix strrchr() implementation
    - clk: socfpga: agilex: fix duplicate s2f_user0_clk
    - csky: don't let si...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.13.0-23.23

---------------
linux (5.13.0-23.23) impish; urgency=medium

  * impish/linux: 5.13.0-23.23 -proposed tracker (LP: #1952263)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync update-dkms-versions helper
    - debian/dkms-versions -- update from kernel-versions (main/2021.11.29)

  * CVE-2021-4002
    - hugetlbfs: flush TLBs correctly after huge_pmd_unshare

  * [SRU][I/OEM-5.13/OEM-5.14] Add MAC passthrough support for more Lenovo docks
    (LP: #1951767)
    - net: usb: r8152: Add MAC passthrough support for more Lenovo Docks

  * Fix non-working e1000e device after resume (LP: #1951861)
    - SAUCE: Revert "e1000e: Additional PHY power saving in S0ix"
    - SAUCE: Revert "e1000e: Add polling mechanism to indicate CSME DPG exit"
    - SAUCE: Revert "e1000e: Add handshake with the CSME to support S0ix"

  * CVE-2021-43267
    - tipc: fix size validations for the MSG_CRYPTO type

  * Impish update: upstream stable patchset 2021-11-22 (LP: #1951880)
    - ext4: check and update i_disksize properly
    - ext4: correct the error path of ext4_write_inline_data_end()
    - ASoC: Intel: sof_sdw: tag SoundWire BEs as non-atomic
    - HID: apple: Fix logical maximum and usage maximum of Magic Keyboard JIS
    - netfilter: ip6_tables: zero-initialize fragment offset
    - HID: wacom: Add new Intuos BT (CTL-4100WL/CTL-6100WL) device IDs
    - ASoC: SOF: loader: release_firmware() on load failure to avoid batching
    - netfilter: nf_nat_masquerade: make async masq_inet6_event handling generic
    - netfilter: nf_nat_masquerade: defer conntrack walk to work queue
    - mac80211: Drop frames from invalid MAC address in ad-hoc mode
    - m68k: Handle arrivals of multiple signals correctly
    - hwmon: (ltc2947) Properly handle errors when looking for the external clock
    - net: prevent user from passing illegal stab size
    - mac80211: check return value of rhashtable_init
    - vboxfs: fix broken legacy mount signature checking
    - net: sun: SUNVNET_COMMON should depend on INET
    - drm/amdgpu: fix gart.bo pin_count leak
    - scsi: ses: Fix unsigned comparison with less than zero
    - scsi: virtio_scsi: Fix spelling mistake "Unsupport" -> "Unsupported"
    - perf/core: fix userpage->time_enabled of inactive events
    - sched: Always inline is_percpu_thread()
    - hwmon: (pmbus/ibm-cffps) max_power_out swap changes
    - ALSA: usb-audio: Unify mixer resume and reset_resume procedure
    - KVM: arm64: nvhe: Fix missing FORCE for hyp-reloc.S build rule
    - pinctrl: qcom: sc7280: Add PM suspend callbacks
    - net: bgmac-platform: handle mac-address deferral
    - scsi: qla2xxx: Fix excessive messages during device logout
    - io_uring: kill fasync
    - upstream stable to v5.10.74, v5.14.13
    - ALSA: usb-audio: Add quirk for VF0770
    - ALSA: pcm: Workaround for a wrong offset in SYNC_PTR compat ioctl
    - ALSA: seq: Fix a potential UAF by wrong private_free call order
    - ALSA: hda/realtek: Enable 4-speaker output for Dell Precision 5560 laptop
    - ALSA: hda - Enable headphone mic on Dell Latitude laptops with ALC3254
    - ALSA: hda/realtek: Complete partial device nam...