emulation of vexpress-a9 vexpress-a15 in multicore mode seems to be broken

[embedded0n3]

Report based on:
https://stackoverflow.com/questions/37068688/have-you-got-issues-with-running-u-boot-for-versatile-express-a9-platform-under

I don't see any bug reports about vexpress boards emulation here, so I decided to let you know:

version tested: qemu 2.0.0 on 32 bit Debian GNU/Linux

There are issues with running u-boot on qemu vexpress-a9 ( it was reported to work ok at physical multicore board ) and vexpress-a15 with parameters -smp cores=2 r 3 or 4.
So far those are two tested. I dont know ATM if any other board emulation works in multiprocessor mode

Original post at stackoverflow:
---------------------------------------------------------------------------------------------------
I have tried numerous releases of u-boot since vexpress-a9 was introduced in u-bot stable release. Under qemu with multicore emulation its behaving unstable - mostly crashes instantly after loading. Sometimes it loads without issues but its rare. My version of qemu is 2.0.0 on x86 32bit Debian. I would be interested to hear if you succeeded in runing it under qemu with multicore emulation ( if yes which version of qemu and u-boot ) or physical versatile express board with multiple cores.

Example outputs are

$ qemu-system-arm -M vexpress-a9 -nographic -kernel u-boot -smp cores=2
U-Boot 2013.01.01-dirty (May 05 2016 - 08:27:19)

DRAM: 128 MiB
WARNING: Caches not enabled
undefined instruction
pc : [<00000000>] lr : [<00000000>]
sp : 60000ef0 ip : 67eedfd8 fp : 00000000
r10: 00000000 r9 : 00000000 r8 : 60000f00
r7 : 00000000 r6 : 608146ac r5 : 60828365 r4 : 0000004d
r3 : 00000000 r2 : 00000000 r1 : 60000f80 r0 : 47cc9e10
Flags: nzcv IRQs on FIQs on Mode USER_26
Resetting CPU ...
resetting ...
Flash: 256 MiB
MMC: MMC: 0
*** Warning - bad CRC, using default environment

In: serial
Out: serial
Err: serial
Net: smc911x-0
Warning: smc911x-0 using MAC address from net device

Hit any key to stop autoboot: 0
Wrong Image Format for bootm command
ERROR: can't get kernel image!
VExpress#

This time it was loaded

Other time it was not...

$ qemu-system-arm -M vexpress-a9 -nographic -kernel u-boot -smp cores=2

U-Boot 2013.01.01-dirty (May 05 2016 - 08:27:19)

DRAM: 128 MiB
WARNING: Caches not enabled

U-Boot 2013.01.01-dirty (May 05 2016 - 08:27:19)

DRAM: 128 MiB
WARNING: Caches not enabled
Flash: 256 MiB
MMC: MMC: 0
*** Warning - bad CRC, using default environment

In: serial
Out: serial
Err: serial
Net: smc911x-0
Warning: smc911x-0 using MAC address from net device

Hit any key to stop autoboot: 2 qemu: fatal: Trying to execute code outside RAM or ROM at 0x6f71c004

R00=00418937 R01=000003e8 R02=00418937 R03=00000000
R04=00000000 R05=67eedf58 R06=67f8e000 R07=00000002
R08=00418937 R09=0778e000 R10=6082f688 R11=00000000
R12=67eedfd8 R13=00000000 R14=67fb82c0 R15=6f71c004
PSR=200001db --C- A und32
s00=00000000 s01=00000000 d00=0000000000000000
s02=00000000 s03=00000000 d01=0000000000000000
s04=00000000 s05=00000000 d02=0000000000000000
s06=00000000 s07=00000000 d03=0000000000000000
s08=00000000 s09=00000000 d04=0000000000000000
s10=00000000 s11=00000000 d05=0000000000000000
vs12=00000000 s13=00000000 d06=0000000000000000
s14=00000000 s15=00000000 d07=0000000000000000
s16=00000000 s17=00000000 d08=0000000000000000
s18=00000000 s19=00000000 d09=0000000000000000
s20=00000000 s21=00000000 d10=0000000000000000
s22=00000000 s23=00000000 d11=0000000000000000
s24=00000000 s25=00000000 d12=0000000000000000
s26=00000000 s27=00000000 d13=0000000000000000
s28=00000000 s29=00000000 d14=0000000000000000
s30=00000000 s31=00000000 d15=0000000000000000
s32=00000000 s33=00000000 d16=0000000000000000
s34=00000000 s35=00000000 d17=0000000000000000
s36=00000000 s37=00000000 d18=0000000000000000
s38=00000000 s39=00000000 d19=0000000000000000
s40=00000000 s41=00000000 d20=0000000000000000
s42=00000000 s43=00000000 d21=0000000000000000
s44=00000000 s45=00000000 d22=0000000000000000
s46=00000000 s47=00000000 d23=0000000000000000
s48=00000000 s49=00000000 d24=0000000000000000
s50=00000000 s51=00000000 d25=0000000000000000
s52=00000000 s53=00000000 d26=0000000000000000
s54=00000000 s55=00000000 d27=0000000000000000
s56=00000000 s57=00000000 d28=0000000000000000
s58=00000000 s59=00000000 d29=0000000000000000
s60=00000000 s61=00000000 d30=0000000000000000
s62=00000000 s63=00000000 d31=0000000000000000
FPSCR: 00000000
-------------------------------------------------------------------------------------------------
end of post

Original discussion at ( includes comments of head custodian of Das U-boot about u-boot working on physical board versatile express a9 and him having issues with runing Linux distro at qemu vexpress-a9 board emulation ):
https://stackoverflow.com/questions/37068688/have-you-got-issues-with-running-u-boot-for-versatile-express-a9-platform-under

[Peter Maydell (pmaydell)]

On 7 May 2016 at 09:08, embedded0n3 <email address hidden> wrote:
> Public bug reported:
>
> Report based on:
> https://stackoverflow.com/questions/37068688/have-you-got-issues-with-running-u-boot-for-versatile-express-a9-platform-under
>
> I don't see any bug reports about vexpress boards emulation here, so I
> decided to let you know:
>
> version tested: qemu 2.0.0 on 32 bit Debian GNU/Linux

This version of QEMU is now five major releases old. Please can
you retry with the most recent the QEMU 2.6.0 release candidate,
or with QEMU master from git and see if your problems are still
present?

thanks
-- PMM

[embedded0n3 (jainformatyka)]

Yes, there are still problems on qemu 2.6.0 release candidate and 2.5.1 with same version of u-boot as mentioned and with latest stable u-boot release. When qemu has parameter -smp cores=2 or 3 or 4.

errors are:
qemu: fatal: Trying to execute code outside RAM or ROM at 0x31306c70
That adress changes but is mostly close to that example value ("8 digits" adress)
as mentioned in previous posted bugreport

Sometimes runing U-boot(?) hangs but I can exit qemu using keys Ctrl-a x
And that seems to be new behaviour.

- - - - - - - - - - - -
If someone is interested how to build u-boot for further testing of this issue on his own box

  $ git clone git://git.denx.de/u-boot.git
  $ cd u-boot
  $ make vexpress_ca9x4_defconfig ARCH=arm CROSS_COMPILE=arm-none-eabi
  $ make all_ca9x4_defconfig ARCH=arm CROSS_COMPILE=arm-none-eabi

[embedded0n3 (jainformatyka)]

Sorry, missed '-' character at the end of command obviously. It shaould be:

  $ make vexpress_ca9x4_defconfig ARCH=arm CROSS_COMPILE=arm-none-eabi-
  $ make all_ca9x4_defconfig ARCH=arm CROSS_COMPILE=arm-none-eabi-

[embedded0n3 (jainformatyka)]

And if this is not clear - in all cases everything was OK when emulation was started without -smp cors=2 or 3 or 4. By OK i mean there was no
 qemu: fatal: Trying to execute code outside RAM or ROM at 0xXXXXXXXX
errors and I was able to execute some comands like help and printenv and nothing happened when qemu was left open for time of 15 minutes ( sometimes in multicoe mulation mode error appeaed after about one and half minute if that time it loaded without almost instant eror message ).

[Peter Maydell (pmaydell)]

Does the u-boot binary you're running expect to work with SMP? Usually "Trying to execute code outside RAM or ROM" means "the guest binary did something badly wrong and jumped off to an invalid address", rather than being a QEMU bug.

[embedded0n3 (jainformatyka)]

Does -smp cores=4 means qemu is emulaing versatile espress a9 with 4 cores ( this is how versatile express board is designed - i mean it's a board with 4 cores cotex a9 ) and qemu without -smp parameter emalates jus single core ( unicore )?

In case of versatile express a9 u-boot binary:

U-boot was compilled from cofig file vexpress_ca9x4 - versatile express cortex a9 4cores

U-boot configuration file is created to run on physical versatile a9 board. And this was confirmed by Tom Rini ( the head custodian of Das U-boot project as I see on his stackoverflow page ) to run without problems on physical 4 core versatile express a9 board ( see stackoverflow disscussion link in 1st message of this bug report ):

Here is citation of his comment:

>I have recently gotten confirmation that the real HW does work on current mainline U-Boot,
> but I can also replicate your failures. Interestingly, the real board is 4 cores,
> and I can (with the kernelci.org Linux kernel / device tree) run with -smp cores=2
> but 3 and 4 result in soft lockups once userland starts. – Tom Rini 2 days ago

So he even had more erors as he get further in linux boot process ( I stopped tests so far on just loading u-boot and waiting few minutes if it will fail suddenly, what happend ). As I said errors were quiet random, and sometimes appeard just after more than minute of inactivity, other times almost instantly after boot ).

[Peter Maydell (pmaydell)]

On 9 May 2016 at 13:47, embedded0n3 <email address hidden> wrote:
> Does -smp cores=4 means qemu is emulaing versatile espress a9 with 4
> cores ( this is how versatile express board is designed - i mean it's a
> board with 4 cores cotex a9 ) and qemu without -smp parameter emalates
> jus single core ( unicore )?

Yes.

> U-boot configuration file is created to run on physical versatile a9
> board. And this was confirmed by Tom Rini ( the head custodian of Das
> U-boot project as I see on his stackoverflow page ) to run without
> problems on physical 4 core versatile express a9 board ( see
> stackoverflow disscussion link in 1st message of this bug report ):
>
> Here is citation of his comment:
>
>>I have recently gotten confirmation that the real HW does work on current mainline U-Boot,
>> but I can also replicate your failures. Interestingly, the real board is 4 cores,
>> and I can (with the kernelci.org Linux kernel / device tree) run with -smp cores=2
>> but 3 and 4 result in soft lockups once userland starts. – Tom Rini 2 days ago
>
> So he even had more erors as he get further in linux boot process ( I
> stopped tests so far on just loading u-boot and waiting few minutes if
> it will fail suddenly, what happend ). As I said errors were quiet
> random, and sometimes appeard just after more than minute of inactivity,
> other times almost instantly after boot ).

The next phase is that somebody needs to debug what the u-boot code is
doing that causes it to crash (this is largely working with the guest
code, not trying to debug QEMU itself).

thanks
-- PMM

[embedded0n3 (jainformatyka)]

I see.
I can take a look.
Could this procedure be helpfull in this case? Error message was similar but it was related to loading coreboot
http://blog.3mdeb.com/2014/08/07/debugging-coreboot-for-qemu-armv7-vexpress-a9-emulated-mainboard/
Or there are better tools to do this job?

[embedded0n3 (jainformatyka)]

I found another typo In comment from last night about compilling u-boot.
This line
 $ make all_ca9x4_defconfig ARCH=arm CROSS_COMPILE=arm-none-eabi-
should look:
 $ make all ARCH=arm CROSS_COMPILE=arm-none-eabi-

[Nicolae Rosia (nicolae.rosia)]

I can confirm this bug with vexpress-a9.
I have tested with the following versions:
QEMU emulator version 2.4.1 (qemu-2.4.1-11.fc23), Copyright (c) 2003-2008 Fabrice Bellard
QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers

Here are the reproducing steps:
Get an ARM cross-compiler, and latest u-boot.
$ cd u-boot
$ make ARCH=arm CROSS_COMPILE=${CROSS_COMPILE} vexpress_ca9x4_defconfig
$ make ARCH=arm CROSS_COMPILE=${CROSS_COMPILE}

# avoid audio errors
$ export QEMU_AUDIO_DRV=none
$ qemu-system-arm -M vexpress-a9 -m 1024 -smp 4 -nographic -kernel u-boot
U-Boot 2016.11-rc2-00090-g5ac5861-dirty (Oct 27 2016 - 14:46:54 +0300)

DRAM: 1 GiB
WARNING: Caches not enabled
Flash:

U-Boot 2016.11-rc2-00090-g5ac5861-dirty (Oct 27 2016 - 14:46:54 +0300)

DRAM: 1 GiB
WARNING: Caches not enabled
Flash:

U-Boot 2016.11-rc2-00090-g5ac5861-dirty (Oct 27 2016 - 14:46:54 +0300)

DRAM: 1 GiB
WARNING: Caches not enabled
Flash:

U-Boot 2016.11-rc2-00090-g5ac5861-dirty (Oct 27 2016 - 14:46:54 +0300)

DRAM: 1 GiB
WARNING: Caches not enabled
Flash: Bad ram pointer (nil)
Aborted (core dumped)

[Nicolae Rosia (nicolae.rosia)]

I believe this is not a qemu boot since booting the kernel with -smp 4 works just fine, sorry for the noise

[Thomas Huth (th-huth)]

Sounds like a U-Boot bug. Closing according to comment #11.