QEMU is not allowing multiple cores with mips architecture

[Skyler Mäntysaari]

I may have found a bug as when trying to boot up an QEMU VM with the following command: "qemu-system-mips -M malta -m 512 -hda hda.img -kernel vmlinux-4.19.0-10-4kc-malta -initrd initrd.img-4.19.0-10-4kc-malta -append "root=/dev/sda1 console=ttyS0" -nographic -device e1000,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::5555-:22 -smp cores=12,threads=1,sockets=1", it will use up all of the CPU cores on the host, but not bootup?

Kernel log also shows:
[ 100.303136] perf: interrupt took too long (2506 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[ 107.656869] perf: interrupt took too long (3195 > 3132), lowering kernel.perf_event_max_sample_rate to 62500
[ 117.668390] perf: interrupt took too long (4033 > 3993), lowering kernel.perf_event_max_sample_rate to 49500
[ 217.166763] perf: interrupt took too long (5126 > 5041), lowering kernel.perf_event_max_sample_rate to 39000
[ 231.910132] perf: interrupt took too long (6445 > 6407), lowering kernel.perf_event_max_sample_rate to 31000
[ 250.170677] perf: interrupt took too long (8087 > 8056), lowering kernel.perf_event_max_sample_rate to 24500
[ 285.391451] perf: interrupt took too long (10126 > 10108), lowering kernel.perf_event_max_sample_rate to 19750
[ 778.588911] perf: interrupt took too long (12770 > 12657), lowering kernel.perf_event_max_sample_rate to 15500
[ 1554.825129] perf: interrupt took too long (15982 > 15962), lowering kernel.perf_event_max_sample_rate to 12500
[ 1622.838910] hrtimer: interrupt took 14758063 ns
[ 1712.932777] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
[ 1712.932780] perf: interrupt took too long (59793 > 19977), lowering kernel.perf_event_max_sample_rate to 3250

System details:

OS: Ubuntu 20.04
QEMU emulator version 4.2.0 (Debian 1:4.2-3ubuntu6.3)

[Christian Ehrhardt  (paelzer)]

Hi Skyler,
mips emulation is rather rare - so there is always a chance that something broke without being recognized at first. For a start let me ask a few questions to corner the issue:

1. did that work before and is a regression in the qemu 4.2 of Ubuntu?
2. check other versions - could you try the same with qemu 3.1 (Bionic) and 5.0 (groovy) of Ubuntu if it is the same?
3. after we know of the above if new/old versions were ok it could be worth checking pristine upstream qemu builds to see if any delta in Ubuntu has to be fixed. Depending on your time and ability to do so would you be able to build from tags of https://git.qemu.org/?p=qemu.git?
(TL;DR; add deb-src lines in /etc/apt/sources.list, apt build-dep qemu;

In general maybe attach the kernel/initrd of your test so others can try the same.
I assume you used something like http://ftp.debian.org/debian/dists/stable/main/installer-mips/current/images/malta/netboot/ ?

Once that is sorted we know better if we look at a "how to use question" or an actual bug - and for the latter if it is a regression of some sort (which would make us hunt for the offending change).

[Christian Ehrhardt  (paelzer)]

I was quickly giving things a check but don't have the time to do the full matrix:

The old bits from https://people.debian.org/~aurel32/qemu/mipsel/
$ qemu-system-mips64el -M malta -kernel vmlinux-3.2.0-4-4kc-malta -nographic -curses

Later the kernels are more obviously split (mipsel/mips64/mips64el) build at https://people.debian.org/~jcowgill/qemu-mips/
I can get to run with:
$ qemu-system-mips64el -M malta -cpu MIPS64R2-generic -m 2G -kernel vmlinux-4.15.0-1-5kc-malta.mips64el.sid -nographic -curses
$ qemu-system-mips64el -M malta -cpu MIPS64R2-generic -m 2G -kernel vmlinux-4.15.0-1-5kc-malta.mipsel.sid -nographic -curses
$ qemu-system-mips64 -M malta -cpu MIPS64R2-generic -m 2G -kernel vmlinux-4.15.0-1-5kc-malta.mips.sid -nographic -curses

Zoning in to something close to your kernel that you've tried from http://ftp.debian.org/debian/dists/stable/main/installer-mips/current/images/malta/netboot/
works with:
$ qemu-system-mips64 -M malta -cpu MIPS64R2-generic -m 2G -kernel vmlinux-4.19.0-10-4kc-malta -nographic -curses

From here I was slowly getting more similar to your commandline.
Sill works:
$ qemu-system-mips -M malta -m 2G -kernel vmlinux-4.19.0-10-4kc-malta -nographic -curses
add initrd:
$ qemu-system-mips -M malta -m 2G -kernel vmlinux-4.19.0-10-4kc-malta -initrd initrd-4.19.0-10-4kc-$malta.gz -nographic -curses
Add net:
$ qemu-system-mips -M malta -m 2G -kernel vmlinux-4.19.0-10-4kc-malta -initrd initrd-4.19.0-10-4kc-malta.gz -nographic -curses -nographic -device e1000,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::5555-:22
reduce memory:
$ qemu-system-mips -M malta -m 512 -kernel vmlinux-4.19.0-10-4kc-malta -initrd initrd-4.19.0-10-4kc-malta.gz -nographic -curses -nographic -device e1000,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::5555-:22
kernel commandline:
$ qemu-system-mips -M malta -m 512 -kernel vmlinux-4.19.0-10-4kc-malta -initrd initrd-4.19.0-10-4kc-malta.gz -nographic -curses -nographic -device e1000,netdev=user.0 -netdev user,id=user.0,hostfwd=tcp::5555-:22 -append "root=/dev/sda1 console=ttyS0"

All that worked, what got it stuck was the smp option that you used.
-smp 1 works
-smp 2 works
-smp 3 stuck
-smp 4 stuck
  qemu-system-mips: /build/qemu-BQ4hMP/qemu-4.2/hw/acpi/cpu.c:198: cpu_hotplug_hw_init: Assertion
  `mc->possible_cpu_arch_ids' failed.
  Aborted (core dumped)
-smp 4 stuck (retry)
[ 0.000000] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: (null)

With more memory I get slightly further until stuck, up to -smp 12 things won't get better.

Not sure how good/bad smp is in emulated mips - we are back to my initial question - did this work in the past and regressed - or is it instead an upstream feature request for (better) mips emu SMP?

[Christian Ehrhardt  (paelzer)]

These are old, but might still be true:
https://lists.gnu.org/archive/html/qemu-devel/2013-06/msg04436.html
https://qemu-devel.nongnu.narkive.com/m6BNDsfR/does-qemu-support-mips-smp2-malta-board
https://<email address hidden>/msg10710.html

If that is it then that is a natural limitation.

The only bit I've seen mentioning >2 to work is
https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

But the I6400 cpu doesn't exist with newer qemus I7200 does but doesn't work either.
This seems like an upstream feature request to me, but one that was mostly denied 7 years ago (by architecture unable to do so).

Let me know if this is a regression and if I missed something.
Otherwise yeah - you could start a new upstream discussion, but no promises it will end any different than the past ones.

[Skyler Mäntysaari (samip537)]

This hasn't worked at any point yet, as I'm trying to emulate MIPS that has more than one core that it can use for a workload.

Images used:
http://ftp.debian.org/debian/dists/stable/main/installer-mips/current/images/malta/netboot/initrd.gz
http://ftp.debian.org/debian/dists/stable/main/installer-mips/current/images/malta/netboot/vmlinux-4.19.0-10-4kc-malta

I don't know if -smp is the right thing to use for it to begin with, but that's the only thing I found that's even close to getting more cores to the system.

My hardware that's running MIPS, is MIPS 1004Kc V2.15, and it has 4 cores. I'm trying to emulate it with QEMU.

-----------------------------------------------------------------------------------------------
processor : 3
cpu model : MIPS 1004Kc V2.15
BogoMIPS : 581.63
wait instruction : yes
microsecond timers : yes
tlb_entries : 32
extra interrupt vector : yes
hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
isa : mips1 mips2 mips32r1 mips32r2
ASEs implemented : mips16 dsp mt
shadow register sets : 1
kscratch registers : 0
package : 0
core : 1
VPE : 1
VCED exceptions : not available
VCEI exceptions : not available
-----------------------------------------------------------------------------------------------

How can I test qemu 5.0 as groovy has not been released yet?

[Christian Ehrhardt  (paelzer)]

Q: How can I test qemu 5.0 as groovy has not been released yet?
A: you can get that already, depending on your preferred way to deploy there are different options:

You can download a daily build of the ISO:
http://cdimage.ubuntu.com/daily-live/current/
http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/

Cloud-images:
https://cloud-images.ubuntu.com/groovy/current/

Or you can upgrade a system that you already have via:
(I assume you do so from 20.04)
set in /etc/update-manager/release-upgrades
prompt=normal
Then run
$ do-release-upgrade -d

Or (least impact to your existing system) use a container.
You only use emulation, so you don't even need tricks to expose x86 HW acceleration for virtualization, just get a 20.10 system and work in there as you'd normally.
$ lxc launch ubuntu-daily:g g-20.10
$ lxc exec g-20.10

[Launchpad Janitor (janitor)]

[Expired for qemu (Ubuntu) because there has been no activity for 60 days.]