[Potential Regression] Unable to create KVM with uvtool on Groovy ARM64

Bug #1929925 reported by Po-Hsu Lin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
Undecided
Unassigned
linux (Ubuntu)
Undecided
Unassigned
Groovy
High
Stefan Bader

Bug Description

Issue found on 5.8.0-54-generic proposed ARM64 kernel with node "helo", "wright"

This issue does not exist in 5.8.0-53.60 so this might be a regression.

Test case:
  1. Deploy Groovy on an ARM64 node
  2. Create a KVM with the following command:
     uvt-kvm create bjf-test release=groovy arch=arm64
     Or you can run the ubuntu_kvm_smoke_test with autotest:
     git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest-client-tests -b master-next
     git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest
     rm -fr autotest/client/tests
     ln -sf ~/autotest-client-tests autotest/client/tests
     sudo apt-get install git python-minimal -y || sudo apt install python2 -y
     AUTOTEST_PATH=/home/ubuntu/autotest sudo -E autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_kvm_smoke_test/control
  3. Install just the proposed kernel and reboot
  4. Repeat step 2

This is 100% reproducible on ARM64, 4 out of 4 attempts (twice on each nodes).

Test log:
  + SUT=bjf-test
  + SSH_KEY=/home/ubuntu/.ssh/id_rsa
  + SSH_OPTIONS='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa'
  ++ lsb_release -c
  ++ awk '{print$2}'
  + DISTRO=groovy
  + ARCHITECTURE=arm64
  + trap cleanup EXIT
  + '[' -z arm64 ']'
  + kvm-ok
  + '[' 0 '!=' 0 ']'
  + '[' '!' -f /home/ubuntu/.ssh/id_rsa ']'
  + '[' arm64 = ppc64el ']'
  ++ uvt-simplestreams-libvirt query
  ++ grep -P 'groovy.*arm64'
  + image='release=groovy arch=arm64 label=daily (20210527)'
  + '[' -z 'release=groovy arch=arm64 label=daily (20210527)' ']'
  + uvt-kvm create bjf-test release=groovy arch=arm64
  uvt-kvm: error: libvirt: unsupported configuration: Emulator '/usr/bin/qemu-system-aarch64' does not support virt type 'kvm'

syslog:
May 28 08:01:07 helo-kernel kernel: [ 1030.945560] mpt3sas 0000:8d:00.0: invalid short VPD tag 00 at offset 1
May 28 08:01:08 helo-kernel kernel: [ 1031.238058] sr 3:0:0:0: [sr0] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.242304] sr 3:0:0:0: [sr0] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.245430] sr 3:0:0:1: [sr1] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.248810] sr 3:0:0:1: [sr1] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.252678] sr 3:0:0:2: [sr2] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.255678] sr 3:0:0:2: [sr2] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.258811] sr 3:0:0:3: [sr3] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.272678] sr 3:0:0:3: [sr3] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:13 helo-kernel systemd[1]: Started Daemon for generating UUIDs.
May 28 08:01:13 helo-kernel kernel: [ 1036.620470] qemu-system-aar using unsupported default IPA limit, upgrade your VMM
May 28 08:01:14 helo-kernel libvirtd[3767]: libvirt version: 6.6.0, package: 1ubuntu3.5 (Christian Ehrhardt <email address hidden> Wed, 07 Apr 2021 13:33:46 +0200)
May 28 08:01:14 helo-kernel libvirtd[3767]: hostname: helo-kernel
May 28 08:01:14 helo-kernel libvirtd[3767]: unsupported configuration: Emulator '/usr/bin/qemu-system-aarch64' does not support virt type 'kvm'

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: linux-image-5.8.0-54-generic 5.8.0-54.61
ProcVersionSignature: Ubuntu 5.8.0-54.61-generic 5.8.18
Uname: Linux 5.8.0-54-generic aarch64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 28 07:04 seq
 crw-rw---- 1 root audio 116, 33 May 28 07:04 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu50.6
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: skip
CurrentDmesg:

Date: Fri May 28 07:10:15 2021
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: To be filled by O.E.M. Saber
PciMultimedia:

ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.8.0-54-generic root=UUID=6660a008-6d09-4dba-a6f8-0fb57ee2582f ro sysrq_always_enabled
RelatedPackageVersions:
 linux-restricted-modules-5.8.0-54-generic N/A
 linux-backports-modules-5.8.0-54-generic N/A
 linux-firmware 1.190.5
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/04/2020
dmi.bios.release: 7.4
dmi.bios.vendor: EDK II
dmi.bios.version: 0ACKL030
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: Saber
dmi.board.vendor: To be filled by O.E.M.
dmi.board.version: 1.0
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 2
dmi.chassis.vendor: To be filled by O.E.M.
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnEDKII:bvr0ACKL030:bd06/04/2020:br7.4:svnTobefilledbyO.E.M.:pnSaber:pvr1.0:rvnTobefilledbyO.E.M.:rnSaber:rvr1.0:cvnTobefilledbyO.E.M.:ct2:cvr1.0:
dmi.product.family: CN99XX
dmi.product.name: Saber
dmi.product.sku: SABER SKU
dmi.product.version: 1.0
dmi.sys.vendor: To be filled by O.E.M.

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

syslog with successful KVM test on 5.8.0-53

description: updated
Po-Hsu Lin (cypressyew)
tags: added: 5.8 kqa-blocker sru-20210510 ubuntu-kvm-smoke-test
Po-Hsu Lin (cypressyew)
description: updated
Po-Hsu Lin (cypressyew)
description: updated
Revision history for this message
Stefan Bader (smb) wrote :

This seems to be a deliberate action caused by:

commit c526c11a462c4f1c0a88735b6e2d1083a9655d0b
Author: Marc Zyngier <email address hidden>
Date: Thu Mar 11 10:00:15 2021 +0000

    KVM: arm64: Reject VM creation when the default IPA size is unsupported

    BugLink: https://bugs.launchpad.net/bugs/1925259

    commit 7d717558dd5ef10d28866750d5c24ff892ea3778 upstream.

    KVM/arm64 has forever used a 40bit default IPA space, partially
    due to its 32bit heritage (where the only choice is 40bit).
    ...

This adds an -EINVAL exit with the following message logged (visible in the rport):
May 28 08:01:13 helo-kernel kernel: [ 1036.620470] qemu-system-aar using unsupported default IPA limit, upgrade your VMM

Revision history for this message
Stefan Bader (smb) wrote :

Maybe missing the following though nothing refers to it:

commit c9b69a0cf0b4336fe7d2e35c46273debc68910fd
Author: Will Deacon <email address hidden>
Date: Fri Sep 11 14:25:29 2020 +0100

    KVM: arm64: Don't constrain maximum IPA size based on host configuration

Revision history for this message
Marcelo Cerri (mhcerri) wrote :

As pointed out by Krzysztof the backport is dropping the line `kvm_ipa_limit = ipa_max;`.

I will prepare a test kernel re-introducing this line.

Revision history for this message
Marcelo Cerri (mhcerri) wrote :

Attaching a patch with the fix to be tested.

I built a test kernel with the patch and the debian packages are available at:

https://kernel.ubuntu.com/~mhcerri/lp1929925.1/

tags: added: patch
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Hi Marcelo,
this kernel work well with the KVM smoke test.

 Running 'sudo -u ubuntu /home/ubuntu/autotest/client/tests/ubuntu_kvm_smoke_test/the-test arm64'
 [stderr] + SUT=bjf-test
 [stderr] + SSH_KEY=/home/ubuntu/.ssh/id_rsa
 [stderr] + SSH_OPTIONS='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa'
 [stderr] ++ lsb_release -c
 [stderr] ++ awk '{print$2}'
 [stderr] + DISTRO=groovy
 [stderr] + ARCHITECTURE=arm64
 [stderr] + trap cleanup EXIT
 [stderr] + '[' -z arm64 ']'
 [stderr] + kvm-ok
 [stdout] INFO: /dev/kvm exists
 [stdout] KVM acceleration can be used
 [stderr] + '[' 0 '!=' 0 ']'
 [stderr] + '[' '!' -f /home/ubuntu/.ssh/id_rsa ']'
 [stderr] + '[' arm64 = ppc64el ']'
 [stderr] ++ uvt-simplestreams-libvirt query
 [stderr] ++ grep -P 'groovy.*arm64'
 [stderr] + image='release=groovy arch=arm64 label=daily (20210527)'
 [stderr] + '[' -z 'release=groovy arch=arm64 label=daily (20210527)' ']'
 [stderr] + uvt-kvm create bjf-test release=groovy arch=arm64
 [stderr] + uvt-kvm wait bjf-test --insecure --ssh-private-key-file /home/ubuntu/.ssh/id_rsa
 [stderr] ++ uvt-kvm ip bjf-test
 [stderr] + ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa ubuntu@192.168.122.110 uname -a
 [stdout] Linux bjf-test 5.8.0-53-generic #60-Ubuntu SMP Thu May 6 07:47:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
 [stderr] ++ cut '-d ' -f 2
 [stderr] +++ uvt-kvm ip bjf-test
 [stderr] ++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa ubuntu@192.168.122.110 lsb_release -c
 [stderr] + RESULT=groovy
 [stderr] + '[' groovy '!=' groovy ']'
 [stderr] + cleanup
 [stderr] + uvt-kvm destroy bjf-test
 [stderr] + '[' arm64 = ppc64el ']'
        GOOD ubuntu_kvm_smoke_test.kvm_smoke_test ubuntu_kvm_smoke_test.kvm_smoke_test timestamp=1622514210 localtime=Jun 01 02:23:30 completed successfully
    END GOOD ubuntu_kvm_smoke_test.kvm_smoke_test ubuntu_kvm_smoke_test.kvm_smoke_test timestamp=1622514210 localtime=Jun 01 02:23:30
 Persistent state client._record_indent now set to 1
 Persistent state client.unexpected_reboot deleted
 END GOOD ---- ---- timestamp=1622514210 localtime=Jun 01 02:23:30

$ uname -a
Linux wright-kernel 5.8.0-54-generic #61+lp1929925.1 SMP Mon May 31 16:19:25 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Thank you.

Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
assignee: nobody → Stefan Bader (smb)
importance: Undecided → High
status: New → In Progress
Changed in linux (Ubuntu):
status: New → Invalid
Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-groovy
Revision history for this message
Sean Feole (sfeole) wrote :

Verified fixed groovy

Verifying base install
           series: groovy
             arch: arm64
        uname -vr: 5.8.0-55-generic #62-Ubuntu SMP Tue Jun 1 08:20:36 UTC 2021

+ SUT=bjf-test
+ SSH_KEY=/root/.ssh/id_rsa
+ SSH_OPTIONS='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /root/.ssh/id_rsa'
++ lsb_release -c
++ awk '{print$2}'
+ DISTRO=groovy
+ ARCHITECTURE=arm64
+ trap cleanup EXIT
+ '[' -z arm64 ']'
+ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
+ '[' 0 '!=' 0 ']'
+ '[' '!' -f /root/.ssh/id_rsa ']'
+ '[' arm64 = ppc64el ']'
++ uvt-simplestreams-libvirt query
++ grep -P 'groovy.*arm64'
+ image='release=groovy arch=arm64 label=daily (20210527)'
+ '[' -z 'release=groovy arch=arm64 label=daily (20210527)' ']'
+ uvt-kvm create bjf-test release=groovy arch=arm64
+ uvt-kvm wait bjf-test --insecure --ssh-private-key-file /root/.ssh/id_rsa
++ uvt-kvm ip bjf-test
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /root/.ssh/id_rsa ubuntu@192.168.122.213 uname -a
Linux bjf-test 5.8.0-53-generic #60-Ubuntu SMP Thu May 6 07:47:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
++ cut '-d ' -f 2
+++ uvt-kvm ip bjf-test
++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /root/.ssh/id_rsa ubuntu@192.168.122.213 lsb_release -c
+ RESULT=groovy
+ '[' groovy '!=' groovy ']'
+ cleanup
+ uvt-kvm destroy bjf-test
+ '[' arm64 = ppc64el ']'

tags: added: verification-done-groovy
removed: verification-needed-groovy
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (51.1 KiB)

This bug was fixed in the package linux - 5.8.0-55.62

---------------
linux (5.8.0-55.62) groovy; urgency=medium

  * groovy/linux: 5.8.0-55.62 -proposed tracker (LP: #1930379)

  * [Potential Regression] Unable to create KVM with uvtool on Groovy ARM64
    (LP: #1929925)
    - SAUCE: KVM: arm64: Assign kvm_ipa_limit

linux (5.8.0-54.61) groovy; urgency=medium

  * groovy/linux: 5.8.0-54.61 -proposed tracker (LP: #1927592)

  * Introduce the 465 driver series, fabric-manager, and libnvidia-nscq
    (LP: #1925522)
    - debian/dkms-versions -- add NVIDIA 465 and migrate 450 to 460

  * linux-image-5.0.0-35-generic breaks checkpointing of container
    (LP: #1857257)
    - SAUCE: overlayfs: fix incorrect mnt_id of files opened from map_files

  * netfilter: x_tables: fix compat match/target pad out-of-bound write
    (LP: #1927682)
    - netfilter: x_tables: fix compat match/target pad out-of-bound write

  * Groovy update: upstream stable patchset 2021-05-04 (LP: #1927150)
    - mt76: fix tx skb error handling in mt76_dma_tx_queue_skb
    - net: fec: ptp: avoid register access when ipg clock is disabled
    - powerpc/4xx: Fix build errors from mfdcr()
    - atm: eni: dont release is never initialized
    - atm: lanai: dont run lanai_dev_close if not open
    - Revert "r8152: adjust the settings about MAC clock speed down for RTL8153"
    - ALSA: hda: ignore invalid NHLT table
    - ixgbe: Fix memleak in ixgbe_configure_clsu32
    - scsi: ufs: ufs-qcom: Disable interrupt in reset path
    - blk-cgroup: Fix the recursive blkg rwstat
    - net: tehuti: fix error return code in bdx_probe()
    - net: intel: iavf: fix error return code of iavf_init_get_resources()
    - sun/niu: fix wrong RXMAC_BC_FRM_CNT_COUNT count
    - cifs: ask for more credit on async read/write code paths
    - gfs2: fix use-after-free in trans_drain
    - cpufreq: blacklist Arm Vexpress platforms in cpufreq-dt-platdev
    - gpiolib: acpi: Add missing IRQF_ONESHOT
    - nfs: fix PNFS_FLEXFILE_LAYOUT Kconfig default
    - NFS: Correct size calculation for create reply length
    - net: hisilicon: hns: fix error return code of hns_nic_clear_all_rx_fetch()
    - net: wan: fix error return code of uhdlc_init()
    - net: davicom: Use platform_get_irq_optional()
    - net: enetc: set MAC RX FIFO to recommended value
    - atm: uPD98402: fix incorrect allocation
    - atm: idt77252: fix null-ptr-dereference
    - cifs: change noisy error message to FYI
    - irqchip/ingenic: Add support for the JZ4760
    - kbuild: add image_name to no-sync-config-targets
    - kbuild: dummy-tools: fix inverted tests for gcc
    - umem: fix error return code in mm_pci_probe()
    - sparc64: Fix opcode filtering in handling of no fault loads
    - habanalabs: Call put_pid() when releasing control device
    - staging: rtl8192e: fix kconfig dependency on CRYPTO
    - u64_stats,lockdep: Fix u64_stats_init() vs lockdep
    - regulator: qcom-rpmh: Correct the pmic5_hfsmps515 buck
    - block: Fix REQ_OP_ZONE_RESET_ALL handling
    - drm/amd/display: Revert dram_clock_change_latency for DCN2.1
    - drm/amdgpu: fb BO should be ttm_bo_type_device
    - drm/radeon: fix AGP dependency
    - nvme: add NVM...

Changed in linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
Po-Hsu Lin (cypressyew)
Changed in ubuntu-kernel-tests:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers