[Potential Regression] Unable to create KVM with uvtool on Groovy ARM64

Bug #1929925 reported by Po-Hsu Lin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
Fix Released
Undecided
Unassigned
linux (Ubuntu)
Invalid
Undecided
Unassigned
Groovy
Fix Released
High
Stefan Bader

Bug Description

Issue found on 5.8.0-54-generic proposed ARM64 kernel with node "helo", "wright"

This issue does not exist in 5.8.0-53.60 so this might be a regression.

Test case:
  1. Deploy Groovy on an ARM64 node
  2. Create a KVM with the following command:
     uvt-kvm create bjf-test release=groovy arch=arm64
     Or you can run the ubuntu_kvm_smoke_test with autotest:
     git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest-client-tests -b master-next
     git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest
     rm -fr autotest/client/tests
     ln -sf ~/autotest-client-tests autotest/client/tests
     sudo apt-get install git python-minimal -y || sudo apt install python2 -y
     AUTOTEST_PATH=/home/ubuntu/autotest sudo -E autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_kvm_smoke_test/control
  3. Install just the proposed kernel and reboot
  4. Repeat step 2

This is 100% reproducible on ARM64, 4 out of 4 attempts (twice on each nodes).

Test log:
  + SUT=bjf-test
  + SSH_KEY=/home/ubuntu/.ssh/id_rsa
  + SSH_OPTIONS='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa'
  ++ lsb_release -c
  ++ awk '{print$2}'
  + DISTRO=groovy
  + ARCHITECTURE=arm64
  + trap cleanup EXIT
  + '[' -z arm64 ']'
  + kvm-ok
  + '[' 0 '!=' 0 ']'
  + '[' '!' -f /home/ubuntu/.ssh/id_rsa ']'
  + '[' arm64 = ppc64el ']'
  ++ uvt-simplestreams-libvirt query
  ++ grep -P 'groovy.*arm64'
  + image='release=groovy arch=arm64 label=daily (20210527)'
  + '[' -z 'release=groovy arch=arm64 label=daily (20210527)' ']'
  + uvt-kvm create bjf-test release=groovy arch=arm64
  uvt-kvm: error: libvirt: unsupported configuration: Emulator '/usr/bin/qemu-system-aarch64' does not support virt type 'kvm'

syslog:
May 28 08:01:07 helo-kernel kernel: [ 1030.945560] mpt3sas 0000:8d:00.0: invalid short VPD tag 00 at offset 1
May 28 08:01:08 helo-kernel kernel: [ 1031.238058] sr 3:0:0:0: [sr0] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.242304] sr 3:0:0:0: [sr0] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.245430] sr 3:0:0:1: [sr1] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.248810] sr 3:0:0:1: [sr1] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.252678] sr 3:0:0:2: [sr2] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.255678] sr 3:0:0:2: [sr2] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.258811] sr 3:0:0:3: [sr3] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:08 helo-kernel kernel: [ 1031.272678] sr 3:0:0:3: [sr3] CDROM not ready. Make sure there is a disc in the drive.
May 28 08:01:13 helo-kernel systemd[1]: Started Daemon for generating UUIDs.
May 28 08:01:13 helo-kernel kernel: [ 1036.620470] qemu-system-aar using unsupported default IPA limit, upgrade your VMM
May 28 08:01:14 helo-kernel libvirtd[3767]: libvirt version: 6.6.0, package: 1ubuntu3.5 (Christian Ehrhardt <email address hidden> Wed, 07 Apr 2021 13:33:46 +0200)
May 28 08:01:14 helo-kernel libvirtd[3767]: hostname: helo-kernel
May 28 08:01:14 helo-kernel libvirtd[3767]: unsupported configuration: Emulator '/usr/bin/qemu-system-aarch64' does not support virt type 'kvm'

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: linux-image-5.8.0-54-generic 5.8.0-54.61
ProcVersionSignature: Ubuntu 5.8.0-54.61-generic 5.8.18
Uname: Linux 5.8.0-54-generic aarch64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 May 28 07:04 seq
 crw-rw---- 1 root audio 116, 33 May 28 07:04 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu50.6
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: skip
CurrentDmesg:

Date: Fri May 28 07:10:15 2021
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: To be filled by O.E.M. Saber
PciMultimedia:

ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.8.0-54-generic root=UUID=6660a008-6d09-4dba-a6f8-0fb57ee2582f ro sysrq_always_enabled
RelatedPackageVersions:
 linux-restricted-modules-5.8.0-54-generic N/A
 linux-backports-modules-5.8.0-54-generic N/A
 linux-firmware 1.190.5
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/04/2020
dmi.bios.release: 7.4
dmi.bios.vendor: EDK II
dmi.bios.version: 0ACKL030
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: Saber
dmi.board.vendor: To be filled by O.E.M.
dmi.board.version: 1.0
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 2
dmi.chassis.vendor: To be filled by O.E.M.
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnEDKII:bvr0ACKL030:bd06/04/2020:br7.4:svnTobefilledbyO.E.M.:pnSaber:pvr1.0:rvnTobefilledbyO.E.M.:rnSaber:rvr1.0:cvnTobefilledbyO.E.M.:ct2:cvr1.0:
dmi.product.family: CN99XX
dmi.product.name: Saber
dmi.product.sku: SABER SKU
dmi.product.version: 1.0
dmi.sys.vendor: To be filled by O.E.M.

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

syslog with successful KVM test on 5.8.0-53

description: updated
Po-Hsu Lin (cypressyew)
tags: added: 5.8 kqa-blocker sru-20210510 ubuntu-kvm-smoke-test
Po-Hsu Lin (cypressyew)
description: updated
Po-Hsu Lin (cypressyew)
description: updated
Revision history for this message
Stefan Bader (smb) wrote :

This seems to be a deliberate action caused by:

commit c526c11a462c4f1c0a88735b6e2d1083a9655d0b
Author: Marc Zyngier <email address hidden>
Date: Thu Mar 11 10:00:15 2021 +0000

    KVM: arm64: Reject VM creation when the default IPA size is unsupported

    BugLink: https://bugs.launchpad.net/bugs/1925259

    commit 7d717558dd5ef10d28866750d5c24ff892ea3778 upstream.

    KVM/arm64 has forever used a 40bit default IPA space, partially
    due to its 32bit heritage (where the only choice is 40bit).
    ...

This adds an -EINVAL exit with the following message logged (visible in the rport):
May 28 08:01:13 helo-kernel kernel: [ 1036.620470] qemu-system-aar using unsupported default IPA limit, upgrade your VMM

Revision history for this message
Stefan Bader (smb) wrote :

Maybe missing the following though nothing refers to it:

commit c9b69a0cf0b4336fe7d2e35c46273debc68910fd
Author: Will Deacon <email address hidden>
Date: Fri Sep 11 14:25:29 2020 +0100

    KVM: arm64: Don't constrain maximum IPA size based on host configuration

Revision history for this message
Marcelo Cerri (mhcerri) wrote :

As pointed out by Krzysztof the backport is dropping the line `kvm_ipa_limit = ipa_max;`.

I will prepare a test kernel re-introducing this line.

Revision history for this message
Marcelo Cerri (mhcerri) wrote :

Attaching a patch with the fix to be tested.

I built a test kernel with the patch and the debian packages are available at:

https://kernel.ubuntu.com/~mhcerri/lp1929925.1/

tags: added: patch
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Hi Marcelo,
this kernel work well with the KVM smoke test.

 Running 'sudo -u ubuntu /home/ubuntu/autotest/client/tests/ubuntu_kvm_smoke_test/the-test arm64'
 [stderr] + SUT=bjf-test
 [stderr] + SSH_KEY=/home/ubuntu/.ssh/id_rsa
 [stderr] + SSH_OPTIONS='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa'
 [stderr] ++ lsb_release -c
 [stderr] ++ awk '{print$2}'
 [stderr] + DISTRO=groovy
 [stderr] + ARCHITECTURE=arm64
 [stderr] + trap cleanup EXIT
 [stderr] + '[' -z arm64 ']'
 [stderr] + kvm-ok
 [stdout] INFO: /dev/kvm exists
 [stdout] KVM acceleration can be used
 [stderr] + '[' 0 '!=' 0 ']'
 [stderr] + '[' '!' -f /home/ubuntu/.ssh/id_rsa ']'
 [stderr] + '[' arm64 = ppc64el ']'
 [stderr] ++ uvt-simplestreams-libvirt query
 [stderr] ++ grep -P 'groovy.*arm64'
 [stderr] + image='release=groovy arch=arm64 label=daily (20210527)'
 [stderr] + '[' -z 'release=groovy arch=arm64 label=daily (20210527)' ']'
 [stderr] + uvt-kvm create bjf-test release=groovy arch=arm64
 [stderr] + uvt-kvm wait bjf-test --insecure --ssh-private-key-file /home/ubuntu/.ssh/id_rsa
 [stderr] ++ uvt-kvm ip bjf-test
 [stderr] + ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa ubuntu@192.168.122.110 uname -a
 [stdout] Linux bjf-test 5.8.0-53-generic #60-Ubuntu SMP Thu May 6 07:47:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
 [stderr] ++ cut '-d ' -f 2
 [stderr] +++ uvt-kvm ip bjf-test
 [stderr] ++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa ubuntu@192.168.122.110 lsb_release -c
 [stderr] + RESULT=groovy
 [stderr] + '[' groovy '!=' groovy ']'
 [stderr] + cleanup
 [stderr] + uvt-kvm destroy bjf-test
 [stderr] + '[' arm64 = ppc64el ']'
        GOOD ubuntu_kvm_smoke_test.kvm_smoke_test ubuntu_kvm_smoke_test.kvm_smoke_test timestamp=1622514210 localtime=Jun 01 02:23:30 completed successfully
    END GOOD ubuntu_kvm_smoke_test.kvm_smoke_test ubuntu_kvm_smoke_test.kvm_smoke_test timestamp=1622514210 localtime=Jun 01 02:23:30
 Persistent state client._record_indent now set to 1
 Persistent state client.unexpected_reboot deleted
 END GOOD ---- ---- timestamp=1622514210 localtime=Jun 01 02:23:30

$ uname -a
Linux wright-kernel 5.8.0-54-generic #61+lp1929925.1 SMP Mon May 31 16:19:25 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Thank you.

Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
assignee: nobody → Stefan Bader (smb)
importance: Undecided → High
status: New → In Progress
Changed in linux (Ubuntu):
status: New → Invalid
Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-groovy
Revision history for this message
Sean Feole (sfeole) wrote :

Verified fixed groovy

Verifying base install
           series: groovy
             arch: arm64
        uname -vr: 5.8.0-55-generic #62-Ubuntu SMP Tue Jun 1 08:20:36 UTC 2021

+ SUT=bjf-test
+ SSH_KEY=/root/.ssh/id_rsa
+ SSH_OPTIONS='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /root/.ssh/id_rsa'
++ lsb_release -c
++ awk '{print$2}'
+ DISTRO=groovy
+ ARCHITECTURE=arm64
+ trap cleanup EXIT
+ '[' -z arm64 ']'
+ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
+ '[' 0 '!=' 0 ']'
+ '[' '!' -f /root/.ssh/id_rsa ']'
+ '[' arm64 = ppc64el ']'
++ uvt-simplestreams-libvirt query
++ grep -P 'groovy.*arm64'
+ image='release=groovy arch=arm64 label=daily (20210527)'
+ '[' -z 'release=groovy arch=arm64 label=daily (20210527)' ']'
+ uvt-kvm create bjf-test release=groovy arch=arm64
+ uvt-kvm wait bjf-test --insecure --ssh-private-key-file /root/.ssh/id_rsa
++ uvt-kvm ip bjf-test
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /root/.ssh/id_rsa ubuntu@192.168.122.213 uname -a
Linux bjf-test 5.8.0-53-generic #60-Ubuntu SMP Thu May 6 07:47:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
++ cut '-d ' -f 2
+++ uvt-kvm ip bjf-test
++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /root/.ssh/id_rsa ubuntu@192.168.122.213 lsb_release -c
+ RESULT=groovy
+ '[' groovy '!=' groovy ']'
+ cleanup
+ uvt-kvm destroy bjf-test
+ '[' arm64 = ppc64el ']'

tags: added: verification-done-groovy
removed: verification-needed-groovy
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (51.1 KiB)

This bug was fixed in the package linux - 5.8.0-55.62

---------------
linux (5.8.0-55.62) groovy; urgency=medium

  * groovy/linux: 5.8.0-55.62 -proposed tracker (LP: #1930379)

  * [Potential Regression] Unable to create KVM with uvtool on Groovy ARM64
    (LP: #1929925)
    - SAUCE: KVM: arm64: Assign kvm_ipa_limit

linux (5.8.0-54.61) groovy; urgency=medium

  * groovy/linux: 5.8.0-54.61 -proposed tracker (LP: #1927592)

  * Introduce the 465 driver series, fabric-manager, and libnvidia-nscq
    (LP: #1925522)
    - debian/dkms-versions -- add NVIDIA 465 and migrate 450 to 460

  * linux-image-5.0.0-35-generic breaks checkpointing of container
    (LP: #1857257)
    - SAUCE: overlayfs: fix incorrect mnt_id of files opened from map_files

  * netfilter: x_tables: fix compat match/target pad out-of-bound write
    (LP: #1927682)
    - netfilter: x_tables: fix compat match/target pad out-of-bound write

  * Groovy update: upstream stable patchset 2021-05-04 (LP: #1927150)
    - mt76: fix tx skb error handling in mt76_dma_tx_queue_skb
    - net: fec: ptp: avoid register access when ipg clock is disabled
    - powerpc/4xx: Fix build errors from mfdcr()
    - atm: eni: dont release is never initialized
    - atm: lanai: dont run lanai_dev_close if not open
    - Revert "r8152: adjust the settings about MAC clock speed down for RTL8153"
    - ALSA: hda: ignore invalid NHLT table
    - ixgbe: Fix memleak in ixgbe_configure_clsu32
    - scsi: ufs: ufs-qcom: Disable interrupt in reset path
    - blk-cgroup: Fix the recursive blkg rwstat
    - net: tehuti: fix error return code in bdx_probe()
    - net: intel: iavf: fix error return code of iavf_init_get_resources()
    - sun/niu: fix wrong RXMAC_BC_FRM_CNT_COUNT count
    - cifs: ask for more credit on async read/write code paths
    - gfs2: fix use-after-free in trans_drain
    - cpufreq: blacklist Arm Vexpress platforms in cpufreq-dt-platdev
    - gpiolib: acpi: Add missing IRQF_ONESHOT
    - nfs: fix PNFS_FLEXFILE_LAYOUT Kconfig default
    - NFS: Correct size calculation for create reply length
    - net: hisilicon: hns: fix error return code of hns_nic_clear_all_rx_fetch()
    - net: wan: fix error return code of uhdlc_init()
    - net: davicom: Use platform_get_irq_optional()
    - net: enetc: set MAC RX FIFO to recommended value
    - atm: uPD98402: fix incorrect allocation
    - atm: idt77252: fix null-ptr-dereference
    - cifs: change noisy error message to FYI
    - irqchip/ingenic: Add support for the JZ4760
    - kbuild: add image_name to no-sync-config-targets
    - kbuild: dummy-tools: fix inverted tests for gcc
    - umem: fix error return code in mm_pci_probe()
    - sparc64: Fix opcode filtering in handling of no fault loads
    - habanalabs: Call put_pid() when releasing control device
    - staging: rtl8192e: fix kconfig dependency on CRYPTO
    - u64_stats,lockdep: Fix u64_stats_init() vs lockdep
    - regulator: qcom-rpmh: Correct the pmic5_hfsmps515 buck
    - block: Fix REQ_OP_ZONE_RESET_ALL handling
    - drm/amd/display: Revert dram_clock_change_latency for DCN2.1
    - drm/amdgpu: fb BO should be ttm_bo_type_device
    - drm/radeon: fix AGP dependency
    - nvme: add NVM...

Changed in linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
Po-Hsu Lin (cypressyew)
Changed in ubuntu-kernel-tests:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.