qemu-i386 user mode can't fork (bash: fork: Invalid argument)

[moonman]

Good time of day everybody,

I have been trying to make usermode qemu on ARM with plugapps (archlinux) with archlinux i386 chroot to work.

1. I installed arch linux in a virtuabox and created a chroot for it with mkarchroot. Transferred it to my pogo plug into /i386/
2. I comiled qemu-i386 static and put it into /i386/usr/bin/
./configure --static --disable-blobs --disable-system --target-list=i386-linux-user
make

3. I also compiled linux kernel 2.6.38 with CONFIG_BINFMT_MISC=y and installed it.
uname -a
Linux Plugbox 2.6.38 #4 PREEMPT Fri Mar 18 22:19:10 CDT 2011 armv5tel Feroceon 88FR131 rev 1 (v5l) Marvell SheevaPlug Reference Board GNU/Linux

4. Added the following options into /etc/rc.local
/sbin/modprobe binfmt_misc
/bin/mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
echo ':qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-i386:' >/proc/sys/fs/binfmt_misc/register

5. Also copied ld-linux.so.3 (actually ld-2.13.so because ld-linux.so.3 is a link to that file) from /lib/ to /i386/lib/

6.Now i chroot into /i386 and I get this:
[root@Plugbox i386]# chroot .
[II aI hnve ao n@P /]# pacman -Suy
bash: fork: Invalid argument

7.I also downloaded linux-user-test-0.3 from qemu website and ran the test:
[root@Plugbox linux-user-test-0.3]# make
./qemu-linux-user.sh
[qemu-i386]
../qemu-0.14.0/i386-linux-user/qemu-i386 -L ./gnemul/qemu-i386 i386/ls -l dummyfile
BUG IN DYNAMIC LINKER ld.so: dl-version.c: 210: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
make: *** [test] Error 127

[moonman (moonman-ca)]

I found some more info about this, however it was on a ppc platform so I don't know if it'll be useful:
http://www.powerdeveloper.org/forums/viewtopic.php?p=13709

"I've made some investigations. Seems that the problem is here:
clone(18874385,0,0,0,1210087208,1074262024) = -1 errno=22 (Invalid argument)
This is a line from qemu-i386 started with -strace option. This error is got by a caller in 3 cases:
1. when child_stack is NULL
2. both CLONE_FS and CLONE_NEWNS flags are set
3. CLONE_THREAD was set, but CLONE_SIGHAND wasn't.
I don't know why there is so many params for that command so I need someone more competent than me to comment the situation"

[moonman (moonman-ca)]

I compiled a static bash build. Now there's no weird command prompt, but I still can't execute anything because of fork.

    [root@Plugbox i386]# chroot .
    bash-44.# ls
    bash: fork: Invalid argument
    bash-44.# a
    bash: fork: Invalid argument
    bash-44.# asdf
    bash: fork: Invalid argument
    bash-44.# asdf;kalsd;fk
    bash: fork: Invalid argument
    bash-44.# exit
    exit
    [root@Plugbox i386]# ldd ./bin/bash
       not a dynamic executable
    [root@Plugbox i386]# file ./bin/bash
    ./bin/bash: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.27, not stripped

[Peter Maydell (pmaydell)]

OK, can you run the following and attach the resulting strace logfiles?

strace -ff -o ls-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/ls

strace -ff -o bash-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/bash -c /bin/ls

(and tell us what the commands print as well).

Also can you state exactly which version of qemu you have compiled? Thanks.

[moonman (moonman-ca)]

Hello,

[root@Plugbox ~]# strace -ff -o ls-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/ls
b? d? e? l? mu-e386i ome oot roc s? u?
bin diae hlrc.tin.gar m? o? oot q s? t? v?

[root@Plugbox ~]# strace -ff -o bash-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/bash -c /bin/ls
/bin/bash: /bin/: %snnca eotcuxe btearinfiy

I compiled version 0.14. I also tried the git version. the same result was the same.

[moonman (moonman-ca)]

Also by googling I noticed that many people have the same problem if their CPU architecture is different from x86, and the try to emulate x86.

[Peter Maydell (pmaydell)]

On 28 March 2011 21:13, moonman <email address hidden> wrote:
> Hello,
>
> [root@Plugbox ~]# strace -ff -o ls-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/ls
> b?   d?    e?            l?  mu-e386i  ome  oot  roc  s?  u?
> bin  diae  hlrc.tin.gar  m?  o?        oot  q    s?   t?  v?
>
> [root@Plugbox ~]# strace -ff -o bash-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/bash -c /bin/ls
> /bin/bash: /bin/: %snnca eotcuxe btearinfiy

Something odd is going on here... Excerpts from the strace:

readlink("roc/self/f", 0x81abf80, 4095) = -1 ENOENT (No such file or directory)

open("/0):/usr/libalo/eNG=Sn_UF.UT!\304\27\10P\254\32\10\267\304\27\10\20/LC_IDENTIFICATION",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/0):/usr/libalo/eNG=Sn_UF.ut/LC_IDENTIFICATION",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/0):/usr/libalo/eNG=Sn_UF/LC_IDENTIFICATION",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/0):/usr/libalo/eNG=Sn.UT!\304\27\10P\254\32\10\267\304\27\10\20/LC_IDENTIFICATION",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/0):/usr/libalo/eNG=Sn.ut/LC_IDENTIFICATION",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/0):/usr/libalo/eNG=Sn/LC_IDENTIFICATION", O_RDONLY|O_LARGEFILE)
= -1 ENOENT (No such file or directory)
write(2, "/bin/bash: /bin/: %snnca eotcuxe"..., 47) = 47

That's clearly an attempt to open something in /proc/self, something
in /usr/lib/locale/, and to print a "cannot execute" message, but
everything's got rather twisted.

Swap every two pairs of bytes (or equivalently, rotate sets of four
characters by two) in this:
: %snnca eotcuxe
...and as if by magic, something comprehensible appears:
%s: cannot execu

Now, running x86 binaries on an ARM host does work for me, but I've
only tested on a Cortex-A8 (ARMv7) host. I think that what's happening
here is that qemu is doing unaligned accesses. On ARMv7 unaligned
accesses "work", ie you get the word you asked for. On ARMv5 the
effect is that the unaligned address is rounded down to a multiple of
four, we load 32 bits and then rotate them -- so you get the effects
you see above.

Short answer: looks like QEMU doesn't currently work on ARMv5 hosts
(although ARMv7 are fine). I'll look into this if I can manage to
scare up some suitable hardware.

[moonman (moonman-ca)]

Wow you are fast!

The hardware I use is pogoplug with plugapps linux. I think any plug computer will do.
http://plugapps.com/index.php5/Install_on_Pogoplug_V2_Pink

They are sold for about $50.

[moonman (moonman-ca)]

Hi,
Just wondering if there has been any progress regarding this issue.
Thanks!

[Peter Maydell (pmaydell)]

I'm afraid not, no.

[moonman (moonman-ca)]

Do you think it will ever get fixed in a reasonable amount of time(or ever) or am I better off just getting an x86 low power board to run x86 binary-only code?

[Peter Maydell (pmaydell)]

> Do you think it will ever get fixed in a reasonable amount of time (or ever)

Well, it's on my list of things to look at, but generally my focus is the current (ARMv7) architecture, and fixing ARMv5 only bugs is lower priority. But there's a pretty good chance I'll get to it somewhere in the next few months.

[Ricardo Padilha (ricardospadilha)]

> But there's a pretty good chance I'll get to it somewhere in the next few months.

That would be amazing. I've got a Linkstation Live v2 and a Drobo FS waiting eagerly for the possibility of running some x86 binaries.

[Oleg (hardhack)]

>> Do you think it will ever get fixed in a reasonable amount of time (or ever)

>Well, it's on my list of things to look at, but generally my focus is the current (ARMv7) architecture, and fixing ARMv5 only bugs is lower priority. But there's a pretty good chance I'll get to it somewhere in the next few months.

Would be awesome to get x86 binaries working on my SheevaPlug. I also think that ARMv5 is more common in devices such as NAS, where it is more feasible to run an emulator like this.
Thanks devs! Waiting eagerly for the fix!

[Andreas Färber (afaerber)]

Peter,

Am 29.03.2011 um 00:09 schrieb Peter Maydell:

> Short answer: looks like QEMU doesn't currently work on ARMv5 hosts
> (although ARMv7 are fine). I'll look into this if I can manage to
> scare up some suitable hardware.

I noticed that configure prints any ARM host as "armv4l". As an
interim measure, should we bump that to armv6l if armv5l is known
broken?

Andreas

[Peter Maydell (pmaydell)]

On 29 May 2011 11:19, Andreas Färber <email address hidden> wrote:
> Am 29.03.2011 um 00:09 schrieb Peter Maydell:
>> Short answer: looks like QEMU doesn't currently work on ARMv5 hosts
>> (although ARMv7 are fine). I'll look into this if I can manage to
>> scare up some suitable hardware.
>
> I noticed that configure prints any ARM host as "armv4l". As an interim
> measure, should we bump that to armv6l if armv5l is known broken?

It's only user-mode that doesn't work, I think, so making configure
barf on earlier systems would be a little harsh (and that still
wouldn't handle the case where you build on one kind of ARM box
and then try to run it on an older one).

It would probably be easier just to fix the bug than figure out
what to do with configure :-)

-- PMM

[moonman (moonman-ca)]

> It's only user-mode that doesn't work, I think

You are correct in saying that only user mode does not work. I did install windows 98 in qemu VM and it worked fine (aside from the fact that it was slow :))

[Peter Maydell (pmaydell)]

I think you should be able to work around this with:
  echo 2 >/proc/cpu/alignment

which makes unaligned accesses fault and the kernel fix them up. This will be slower than if qemu wasn't generating unaligned accesses in the first place but it should at least make things run. (echo 0 ... will give you the original behaviour back.)

(I have an armv5 system running now.)

[Oleg (hardhack)]

Hmm, it does not seem to work for me:

[root@Plugbox cpu]# cat alignment
User: 73412
System: 1
Skipped: 0
Half: 5312
Word: 68091
DWord: 1
Multi: 0
User faults: 2 (fixup)
[root@Plugbox cpu]# cd /i386/
[root@Plugbox i386]# chroot .
bash-4.2# pwd
/
bash-4.2# pacman -Scc
bash: fork: Invalid argument
bash-4.2# ls
bash: fork: Invalid argument
bash-4.2# top
bash: fork: Invalid argument
bash-4.2#

[moonman (moonman-ca)]

Neither does it work for me:

[root@Plugbox /]# cat /proc/cpu/alignment
User: 293831
System: 1
Skipped: 0
Half: 22200
Word: 271546
DWord: 1
Multi: 0
User faults: 2 (fixup)
[root@Plugbox /]# chroot /i386
[root@Plugbox /]# ls
bash: fork: Invalid argument

[moonman (moonman-ca)]

I have to add that it's with kernel 2.6.39 and qemu 0.14.1 compiled from source

[root@Plugbox ~]# uname -a
Linux Plugbox 2.6.39-ARCH #1 PREEMPT Tue Jun 14 15:55:01 MDT 2011 armv5tel Feroceon 88FR131 rev 1 (v5l) Marvell SheevaPlug Reference Board GNU/Linux
[root@Plugbox ~]# /i386/usr/bin/qemu-i386 -h
qemu-i386 version 0.14.1, Copyright (c) 2003-2008 Fabrice Bellard
usage: qemu-i386 [options] program [arguments...]
Linux CPU emulator (compiled for i386 emulation)
...

[Ricardo Padilha (ricardospadilha)]

Same here. The "workaround" does not work on a DroboFS, since it was actually the default setting all along.

root@DroboFS:~# cat /proc/cpu/alignment
User: 74283231
System: 0
Skipped: 0
Half: 0
Word: 74283231
DWord: 0
Multi: 0
User faults: 2 (fixup)

root@DroboFS:~# uname -a
Linux DroboFS 2.6.22.18 #1 Thu Jan 20 14:29:47 PST 2011 armv5tejl GNU/Linux

Tested qemu 0.14.0, but I can give 0.14.1 a try as well, if necessary.

[Peter Maydell (pmaydell)]

OK, that's pretty conclusive. My initial theory about what's going on here can't be right -- I'll investigate further.

[moonman (moonman-ca)]

Did you modify some code to get it working on your armv5 ?

[Peter Maydell (pmaydell)]

No, I just wrote a simple test program to check the thing I thought was the problem (unaligned accesses). Do you still see the mangled text problems you describe earlier in the bug report as well as the 'fork: invalid argument' problem? I've reproduced the fork error message, but not the mangled text, and I'm having trouble thinking how that text mangling could happen with the /proc/cpu/alignment set to 2. (So I'm wondering if we actually have two bugs here.)

To be concrete: does this still generate garbled text output?
 strace -ff -o ls-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/ls
Does this?
 chroot /i386 /usr/bin/qemu-i386 /bin/ls

I'll look at the fork issue, anyway.

[moonman (moonman-ca)]

Garbled text might have been fixed in the recent versions as it seems. Even with User faults == 0 the text is not garbled.

[root@Plugbox ~]# strace -ff -o ls-strace.log chroot /i386 /usr/bin/qemu-i386 /bin/ls
Inconsistency detected by ld.so: dl-version.c: 230: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
[root@Plugbox ~]# cat /proc/cpu
cpu/ cpuinfo
[root@Plugbox ~]# cat /proc/cpu/alignment
User: 23
System: 1
Skipped: 0
Half: 0
Word: 0
DWord: 1
Multi: 0
User faults: 0 (ignored)

 strace -ff -o ls-strace-alignment-2.log chroot /i386 /usr/bin/qemu-i386 /bin/ls
qemu: Unsupported syscall: 240
bin dev home media opt root srv tmp var
boot etc lib mnt proc sbin sys usr
[root@Plugbox ~]# cat /proc/cpu/alignment
User: 23
System: 1
Skipped: 0
Half: 0
Word: 0
DWord: 1
Multi: 0
User faults: 2 (fixup)

 chroot /i386 /usr/bin/qemu-i386 /bin/ls
qemu: Unsupported syscall: 240
bin dev home media opt root srv tmp var
boot etc lib mnt proc sbin sys usr

[moonman (moonman-ca)]

[root@Plugbox ~]# strace -ff -o bash-strace-alignment-2.log chroot /i386 /usr/bin/qemu-i386 /bin/bash
[root@Plugbox /]# ls
bash: fork: Invalid argument
[root@Plugbox /]# exit
exit

[moonman (moonman-ca)]

Ok, it seemed as though "ls" worked from "strace -ff -o ls-strace-alignment-2.log chroot /i386 /usr/bin/qemu-i386 /bin/ls"

so I tried to execute the package manager:

[root@Plugbox ~]# chroot /i386 /usr/bin/qemu-i386 /usr/bin/pacman -Suy qemu: Unsupported syscall: 240
:: Synchronizing package databases...
qemu: Unsupported syscall: 240
qemu: Unsupported syscall: 240
 core 37.0K 166.6K/s 00:00:01 [######################] 100%
 extra 466.9K 524.3K/s 00:00:01 [######################] 100%
 community 446.5K 518.5K/s 00:00:01 [######################] 100%
:: The following packages should be upgraded first :
    pacman
:: Do you want to cancel the current operation
:: and upgrade these packages now? [Y/n] Y

resolving dependencies...
looking for inter-conflicts...

Targets (1): pacman-3.5.3-1

Total Download Size: 0.82 MB
Total Installed Size: 2.78 MB

Proceed with installation? [Y/n] Y
:: Retrieving packages from core...
 pacman-3.5.3-1-i686 840.0K 731.3K/s 00:00:02 [######################] 100%
checking package integrity...
(1/1) checking for file conflicts [######################] 100%
(1/1) upgrading pacman [######################] 100%
warning: /etc/pacman.conf installed as /etc/pacman.conf.pacnew
error: could not fork a new process (Invalid argument)
error: could not fork a new process (Invalid argument)

Still there's something going on with fork

[moonman (moonman-ca)]

If you need an i386 chroot for testing, I created one a while ago (archlinux): http://www.mediafire.com/download.php?qq63mnay2byqqie

[Daniel Schell (danielschell)]

I hope this can work so development for Dropbox on Drobo-FS can continue....

[Artyom (theartlav)]

I have the same exact problem on MIPSel host as well.

Tried Qemu 0.14.1 and git from 110724.
Trying to run something from bash gives fork: Invalid argument.
strace gives clone(18874385,0,0,0,1084164376,1082144312) = -1 errno=22 (Invalid argument)
There is no alignment switches on this CPU.

So, it seems to be impossible to do useful user mode emulation of x86 from non-x86 platforms, which essentially makes the whole user mode emulation defunct.

[Artyom (theartlav)]

Further research shown that the issue is that NPTL (Native Posix Threads) is not supported for i386 user mode emulation.
That causes the described issue in binaries compiled with NPTL (most modern ones).

One way to fix this that worked for me is a patch from here:
http://patchwork.ozlabs.org/patch/45206

I was able to chroot into i386 on mipsel host with this one, and it mostly worked, no problems running other programs from bash.

[moonman (moonman-ca)]

Awesome Artyom! I'm compiling now. I will report back if this patch works for arm as well.

[moonman (moonman-ca)]

YES! That did the trick!

Though, with this patch I was unable to compile qemu 0.15rc1. I suppose the patch wasn't meant for it, but with 0.14.1 it works flawlessly.

Also, I had to do echo 2 >/proc/cpu/alignment as was mentioned by Peter. I hope the patch gets applied to the main tree.

[Justin Shafer (justinshafer)]

I would like to run wine. I have compiled qemu 0.15.5 from git and I noticed we can enable nptl without a patch. I can run an x86 nptl bash but I havent tried to chroot. I have the posix wine running with qemu... So i tried it with wine and I am able to get a --version out of it, but when I actually run it, it does a fork. It will ask for library files and I will stick them in /usr/gnemul/qemu-i386/usr/lib one at a time and everntually it stops asking for files and just errors out.

Tried 0.9.22, 1.01 and 1.21... Stole files from Ubuntu 5, Ubuntu 9, etc.

root@localhost:/wine/usr/lib# wine-pthread
wine-pthread
qemu: Unsupported syscall: 240
Usage: wine PROGRAM [ARGUMENTS...] Run the specified program
       wine --help Display this help and exit
       wine --version Output version information and exit

root@localhost:/wine/usr/lib# wine-pthread notepad.exe
wine-pthread notepad.exe
qemu: Unsupported syscall: 240
qemu: Unsupported syscall: 240
wine: fork : Invalid argument

On Posix it does an unsuportive syscall of 254 but works. I have seen it do a 240 though, but thats when its asking for library files.. I cant help but wonder if its needing a file, though not asking me.

[Justin Shafer (justinshafer)]

Here is a strace.. fumex calls???

strace -ff -o /ls-strace-alignment-6.log /usr/bin/qemu-i386 /usr/bin/wine-pthread notepad.exe

[moonman (moonman-ca)]

I had exactly the same behavior with a patched 0.14.1. Wine doesn't work and that was the reason I was so eager to run qemu.

[Justin Shafer (justinshafer)]

Moonman, you know what I realized. I have the posix version of wine running okay with qemu-i386... But I have never been able to compile my own working version.. I realized that last night. I can compile, and it will run, but wine never actually runs, and Im compiling with --enable-sdl and all that. Tonight I am gonna try --cpu=armv7el instead of the armv4

[Justin Shafer (justinshafer)]

See this can run the posix wine.. But I cant recreate it. Its running qemu-0.12.2 which you can download and compile but I still cant get it to play solitaire like I can with the one below.

wget http://a.trap.me.uk/qemu-i386

[Justin Shafer (justinshafer)]

Okay I found that 12.5 I can run posix wine and that is compiling on my own. gonna try the patch on 12.5 to see if it can run dynamic nptl binaries

[Justin Shafer (justinshafer)]

Odd.. If I do ./configure --static --enable-sdl --target-list=i386-linux-user this will work fine with 12.5 but on say 15 it cant find SDL. I am on 1.2.13 I think.

[Justin Shafer (justinshafer)]

https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/855630

I decided to give it, its own bug.

[Jeffry Johnston (ubuntu-kidsquid)]

Just wanted to mention that the patch in comment #32 worked for me also. I had to use the qemu-0.14.1.tar.gz source as mentioned in comment #34. On 0.15.0 I did not get a compile error, but when it ran I still couldn't fork.

[Justin Shafer (justinshafer)]

Something odd. I installed VMWare with Ubuntu 11.04. Then I installed wine and qemu with apt-get. Same exact errors using qemu-i386. Verbatim.

[Justin Shafer (justinshafer)]

No armel at all in #44. Perhaps if qemu cant run wine with user emulation using x86.. it will not be able to do arm as well?

[Steve (svrusso1)]

This bug also pertains to armv7l. I have a Debian chroot install on my Android phone, and was trying to do an i386 chroot within my chroot (want to run the i386 specific binaries included in the Android SDK).

I installed the qemu-user-static package (tried both v 0.14.1 and 0.15.0).

When putting qemu-i386-static in the usr/bin folder of my chroot, and trying to run `chroot .` I got the fork error mentioned above when executing any command-line programs. Incidentally-- it *did* work if I used `exec ls`, but obviously that would kill my chroot. I assume using exec just circumvented the fork() call.

I was able to native compile qemu 0.14.1 as --static, with the patch provided in #32 above, and I can confirm it works fine -- no fork error.

@Peter Maydell -- do you think you will get that patch incorporated into the main fork of QEMU? I imagine it becoming a bigger issue as more and more people start running full Linux distros on their ARM-based phones; especially as the mobile processors get faster.

[Peter Maydell (pmaydell)]

Steve: there are a number of issues with that patch:
 * x86 cpu_set_tls() doesn't belong in linux-user/syscall.c (but it's not trivial to put it in target-i386 because it's calling do_set_thread_area())
 * it's not "obviously correct" and the author says it needs review, and I'd have to dig out info about this obscure corner of the x86 ABI/architecture
 * I'm pretty sure it's not the only thing needed for threading support on x86, so (until/unless I look much more closely at the whole area) I don't have much confidence that this patch is a coherent single part of the required work
 * there's no Signed-off-by: line from the author so it can't be committed as is

Hopefully somebody else on the list will have time to look properly at the patch; I'm afraid I don't expect to currently.

[Ricardo Padilha (ricardospadilha)]

Just to give some support to what Peter said, here is my experience with the patch from #32.

I cross-compiled qemu 0.14.1 with the patch to ARMv5 and tried to run the i386 linux binary for dropbox. Although I no longer see the fork error message, the process gets stuck in an infinite loop running at 100% CPU. When started with -strace, this is where it gets stuck:

clock_gettime(0,1169900288,135693248,1,0,1169900372) = 0
futex(0x081683c4,FUTEX_PRIVATE_FLAG|FUTEX_WAIT,1,0x45bb4300,0x081683f0,135693296​) = -1 errno=110 (Connection timed out)
gettimeofday(1169900364,0,0,1,142600008,1169900392) = 0
futex(0x081683f0,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,1,0x081683f0,0x087fe748,142600008​) = 0

This four line block gets repeated over and over again.

To the best of my knowledge, this is the same as the dreaded "unsupported syscall 240" error, just with fancier output. My feeling is that the patch only fixes some very small part of the problem.

[Steve (svrusso1)]

@Peter - thanks, fair enough... I don't know enough about qemu's source to understand what you mean, but clearly it's a complex issue. For now the patch seems to work for me-- I haven't had the issue that @Ricardo discusses (again, I'm on armv7l, though).

[moonman (moonman-ca)]

@Ricardo I did not have such a problem and it ran all simple programs just fine (top, ls, cd, cp and so on) on my armv5t. Though I did not cross-compile, but natively compiled qemu on the box itself.
It would be cool to have it working, but I don't think it will happen any time soon. I just bought eeepc 700 for $50 and run all x86 binary-only stuff on it as full-qemu is too heavy :)

[Justin Shafer (justinshafer)]

HOW TO COMPILE ON ARM AND UBUNTU 12.04

This will run wine if you compile with 0.13 or 0.14

sudo apt-get install zlib1g-dev
sudo apt-get install libsdl1.2-dev
./configure --target-list=i386-linux-user --enable-sdl --prefix=/usr --cross-prefix=arm-linux-gnueabi- --host-cc=gcc4.6 --extra-cflags=-marm --cpu=armv4l
make
sudo make install

Works great on Droid4 with Ubuntu 12.04 Chroot and Wine 0.9.14
http://www.onsitedentalsystems.com/wine.tar.gz
DOES NOT WORK ON 0.15.. NO IDEA WHY.

[Justin Shafer (justinshafer)]

how to run without binfmt support in the kernel.

/usr/bin/qemu-i386 /usr/bin/wineserver
/usr/bin/qemu-i386 /usr/bin/wine-pthread notepad.exe
or
/usr/bin/qemu-i386 -L /usr/gnemul/qemu-i386 /usr/bin/wineserver
/usr/bin/qemu-i386 -L /usr/gnemul/qemu-i386 /usr/bin/wine-pthread notepad.exe

If you do have binfmt support.

mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
echo ':i386:M::\x7fELF\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff:/home/user/qemu-i386:' >/proc/sys/fs/binfmt_misc/register

then you can just do
/usr/bin/qemu-i386 /usr/bin/wine-pthread notepad.exe
everytime.

[Michal Suchanek (hramrach)]

So this is a compiler or system header error?

Anybody examined the differences in code generated with native compiler and crosscompiler?

Is there some code difference or are wrong kernel headers pulled from i386 resulting in invalid syscalls?

[Peter Maydell (pmaydell)]

Michal Suchanek wrote:
> So this is a compiler or system header error?
>
> Anybody examined the differences in code generated with native compiler and crosscompiler?

...this comment doesn't make much sense to me -- did you add it to the wrong bug report by mistake? i386 user mode's issues are not related to any cross-vs-native compilation issue or to system header mismatches. It is straightforwardly missing functionality in QEMU that causes various effects which manifest differently depending on what exactly the target binary is doing and how it was compiled (eg which target i386 libc).

[Michal Suchanek (hramrach)]

It might be just coincidence but if you read the above comments you will notice that people who compiled qemu natively on arm report that they can run various binaries without issue but people who crosscompiled report that they can't run the simplest i386 shell utilities.

How come that the functionality that is missing magically appears for some people?

[Justin Shafer (justinshafer)]

I do now know... I compile on my phone and tablet. In fact I loaded Ubuntu on my Touchpad last night, native.

http://code.google.com/p/hp-touchpad-ubuntu/wiki/Installation

No more chroot to deal with.

[Peter Maydell (pmaydell)]

> How come that the functionality that is missing magically appears for some people?

Coincidence. Nobody on this bug report has reported that they've been able to run x86 binary X with a native compiled qemu but not with a cross compiled version of the same qemu sources. I think it is vastly more likely that the people who find that some of their code works are using differently compiled x86 binaries which happen not to use a glibc which provokes the 'invalid syscall' issue [ie fork() does not end up doing a clone syscall with whatever flags are causing us to return EINVAL].

[Justin Shafer (justinshafer)]

I am now able to run winecfg... you have to have wine-pthread run winecfg
qemu-i386 /usr/bin/qemu-i386 wine-pthread winecfg
All the tabs load except audio.. For that it hangs..

Trying to fix that.. I want to run an app that hangs and it uses audio.

[Justin Shafer (justinshafer)]

Interesting stuff.
With 0.14 and 1.2
wineserver will run if you say wineserver -d2 -f -p for example.
I believe it is forking when you run plain old wineserver because it really is getting an invalid argument.

I am running Wine 1.1.14 and Qemu 0.14 and I can run many apps.

I cannot run a NeoBook app..

"Runtime error 216 at 004040E6"

Any idea why? =)

If you run Wine 1.1.14 and the latest qemu from master as of tonight.. wineserver will load with wine-pthread but when wine-pthread runs you get "connection reset by peer" by wine-pthread. Just an FYI

Wine 1.1.4 was taken from Fedora Cora 9

[Justin Shafer (justinshafer)]

Does this have anything to do with Memory Mapping? I have a feeling it does....

I notice the mmap_min_addr setting for arm is 4096

For x86 its 65536

http://www.onsitedentalsystems.com/error.jpg

[Justin Shafer (justinshafer)]

The app I am testing does run on qemu-i386 compiled for x86.. and the host is x86.

The screen shot I posted in the last post is what happens when you run qemu and wine on arm instead of qemu and wine on x86.

[Justin Shafer (justinshafer)]

Requirements to get Wine 1.5.11 and Qemu to work with Ubuntu 12.10 on ARM

1. VMSPLIT-3G and BINFMT_MISC must be compiled into the kernel.. It makes my kernel crash when I access the lan\wifi with traffic..
2. Use Qemu-0.14.1 with the NPTL Patch ./configure --enable-sdl --target-list=i386-linux-user --prefix=/usr --extra-cflags=-marm to compile.. I compile it in Ubuntu for Arm.
3. Compile Wine 1.5.11 on Ubuntu 12.10 32bit X86.
4. Create /usr/gnemul/qemu-i386/lib /usr/gnemul/qemu-i386/usr/local/lib, /usr/gnemul/qemu-i386/usr/local/lib/wine /usr/gnemul/qemu-i386/usr/lib /usr/gnemul/qemu-i386/usr/lib/i386-linux-gnu and copy the corresponding files from X86. Qemu\Wine will ask for them. Bring over your ~/.wine directory and put it in your home folder.
5. sudo apt-get build-dep wine (do this on arm)
6 Binfmt time. Here is my script to get wine to run
echo ':i386:M::\x7fELF\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff:/usr/bin/qemu-i386:' >/proc/sys/fs/binfmt_misc/register
echo ':DOSWin:M::MZ::/usr/local/bin/wine:' >/proc/sys/fs/binfmt_misc/register

7. If you do ALL that... Then winecfg will run. About to start testing programs.

Pinball runs VERY FAST! VERY PLAYABLE NOW!!!!!!!!!!! :D

Justin Shafer
OnsiteDentalSystems.com

[Justin Shafer (justinshafer)]

Oh yeah.. wine goes in /usr/local/bin. I created a symbolic link from /usr/gnemul/qemu-i386/usr/local/lib/libwine* /usr/local/lib. Same with the wine folder.. it needs to be seen as /usr/local/lib/wine.

[Justin Shafer (justinshafer)]

I still have one program that refuses to run.. I think its having a problem with an x86 library on the gnemul side... I noticed libpng.so.3 wanted to be in the gnemul folder by wine.. even though it didn't exist on the X86 machine I was using to compile.. so I copied it from libpng12.so..

Anyone have an Idea about that???????????

[Justin Shafer (justinshafer)]

Something new...

I have started to compile qemu with all the audio drivers --audio-drv-list="oss alsa ps sdl esd" on 0.14.1 with a patch that was included in 0.15 and it compiles IF I edit ioctls.h and remove 3 lines about sound. Oddly enough my kernel from CM9\Ubuntu is not compiled with those 3 lines I removed.. So I am recompiling my kernel and then I am going to recompile qemu and I bet I can compile it without removing those 3 lines.

-Justin

Wine 1.5.11 is a lot funner to play with then 0.9.14

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Bug Watch Updater
Sent: Sunday, November 18, 2012 5:39 AM
To: <email address hidden>
Subject: [Bug 739785] Re: qemu-i386 user mode can't fork (bash: fork: Invalid argument)

** Changed in: qemu (Debian)
       Status: Unknown => Confirmed

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/739785

Title:
  qemu-i386 user mode can't fork (bash: fork: Invalid argument)

Status in QEMU:
  New
Status in “qemu” package in Debian:
  Confirmed

Bug description:
  Good time of day everybody,

  I have been trying to make usermode qemu on ARM with plugapps
  (archlinux) with archlinux i386 chroot to work.

  1. I installed arch linux in a virtuabox and created a chroot for it with mkarchroot. Transferred it to my pogo plug into /i386/
  2. I comiled qemu-i386 static and put it into /i386/usr/bin/
  ./configure --static --disable-blobs --disable-system --target-list=i386-linux-user
  make

  3. I also compiled linux kernel 2.6.38 with CONFIG_BINFMT_MISC=y and installed it.
  uname -a
  Linux Plugbox 2.6.38 #4 PREEMPT Fri Mar 18 22:19:10 CDT 2011 armv5tel Feroceon 88FR131 rev 1 (v5l) Marvell SheevaPlug Reference Board GNU/Linux

  4. Added the following options into /etc/rc.local
  /sbin/modprobe binfmt_misc
  /bin/mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
  echo ':qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-i386:' >/proc/sys/fs/binfmt_misc/register

  5. Also copied ld-linux.so.3 (actually ld-2.13.so because ld-
  linux.so.3 is a link to that file) from /lib/ to /i386/lib/

  6.Now i chroot into /i386 and I get this:
  [root@Plugbox i386]# chroot .
  [II aI hnve ao n@P /]# pacman -Suy
  bash: fork: Invalid argument

  7.I also downloaded linux-user-test-0.3 from qemu website and ran the test:
  [root@Plugbox linux-user-test-0.3]# make
  ./qemu-linux-user.sh
  [qemu-i386]
  ../qemu-0.14.0/i386-linux-user/qemu-i386 -L ./gnemul/qemu-i386 i386/ls -l dummyfile
  BUG IN DYNAMIC LINKER ld.so: dl-version.c: 210: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
  make: *** [test] Error 127

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/739785/+subscriptions

[Justin Shafer (justinshafer)]

Open GL Works with wglgears.

ubuntu@hptp-u1210b1:~/.wine/drive_c$ wine ./wglgears.exe
qemu: Unsupported syscall: 254
Unsupported ancillary data: 1/2
qemu: Unsupported syscall: 242
Unsupported ancillary data: 1/2
qemu: Unsupported syscall: 242
Unsupported ancillary data: 1/2
qemu: Unsupported syscall: 242
Unsupported ancillary data: 1/2
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
Unsupported ancillary data: 1/2
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
Unsupported ancillary data: 1/2
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
qemu: Unsupported syscall: 242
Unsupported ancillary data: 1/2
qemu: Unsupported syscall: 242
err:winediag:X11DRV_WineGL_InitOpenglInfo Direct rendering is disabled, most likely your OpenGL drivers haven't been installed correctly (using GL renderer "Software Rasterizer", version "1.4 (2.1 Mesa 9.0)").
366 frames in 5.0 seconds = 73.200 FPS
315 frames in 5.0 seconds = 63.000 FPS
312 frames in 5.0 seconds = 62.400 FPS
334 frames in 5.0 seconds = 66.800 FPS
340 frames in 5.0 seconds = 68.000 FPS
320 frames in 5.0 seconds = 64.000 FPS
340 frames in 5.0 seconds = 68.000 FPS
320 frames in 5.0 seconds = 64.000 FPS
340 frames in 5.0 seconds = 68.000 FPS
340 frames in 5.0 seconds = 68.000 FPS

[James Le Cuirot (chewi)]

I have just encountered this trying to emulate i386 on x86_64, which should dismiss any theories about ARM or MIPS. I've tried to apply the previous patch to QEMU 1.2.2 but it doesn't build. Currently trying to fix it.

[James Le Cuirot (chewi)]

I get an undefined reference to cpu_set_tls. The other architectures have this defined in target-*/cpu.h but the implementations vary. They generally seem to modify a register or two. I'm out of my depth here. I have no idea what that would look like for i386.

[James Le Cuirot (chewi)]

Apologies, I missed part of the patch when trying to reapply it. Here it is. It seems to work.

[Justin Shafer (justinshafer)]

http://forum.winehq.org/viewtopic.php?f=2&t=17701

Here is where I got... Read the whole thing..

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of James Le Cuirot
Sent: Monday, January 21, 2013 4:16 PM
To: <email address hidden>
Subject: [Bug 739785] Re: qemu-i386 user mode can't fork (bash: fork: Invalid argument)

Apologies, I missed part of the patch when trying to reapply it. Here it is. It seems to work.

** Patch added: "add-usermode-NPTL-support-for-i386.patch"
   https://bugs.launchpad.net/qemu/+bug/739785/+attachment/3493200/+files/add-usermode-NPTL-support-for-i386.patch

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/739785

Title:
  qemu-i386 user mode can't fork (bash: fork: Invalid argument)

Status in QEMU:
  New
Status in “qemu” package in Debian:
  Confirmed

Bug description:
  Good time of day everybody,

  I have been trying to make usermode qemu on ARM with plugapps
  (archlinux) with archlinux i386 chroot to work.

  1. I installed arch linux in a virtuabox and created a chroot for it with mkarchroot. Transferred it to my pogo plug into /i386/
  2. I comiled qemu-i386 static and put it into /i386/usr/bin/
  ./configure --static --disable-blobs --disable-system --target-list=i386-linux-user
  make

  3. I also compiled linux kernel 2.6.38 with CONFIG_BINFMT_MISC=y and installed it.
  uname -a
  Linux Plugbox 2.6.38 #4 PREEMPT Fri Mar 18 22:19:10 CDT 2011 armv5tel Feroceon 88FR131 rev 1 (v5l) Marvell SheevaPlug Reference Board GNU/Linux

  4. Added the following options into /etc/rc.local
  /sbin/modprobe binfmt_misc
  /bin/mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
  echo ':qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-i386:' >/proc/sys/fs/binfmt_misc/register

  5. Also copied ld-linux.so.3 (actually ld-2.13.so because ld-
  linux.so.3 is a link to that file) from /lib/ to /i386/lib/

  6.Now i chroot into /i386 and I get this:
  [root@Plugbox i386]# chroot .
  [II aI hnve ao n@P /]# pacman -Suy
  bash: fork: Invalid argument

  7.I also downloaded linux-user-test-0.3 from qemu website and ran the test:
  [root@Plugbox linux-user-test-0.3]# make
  ./qemu-linux-user.sh
  [qemu-i386]
  ../qemu-0.14.0/i386-linux-user/qemu-i386 -L ./gnemul/qemu-i386 i386/ls -l dummyfile
  BUG IN DYNAMIC LINKER ld.so: dl-version.c: 210: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
  make: *** [test] Error 127

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/739785/+subscriptions

[Peter Maydell (pmaydell)]

This is a long and confusing bug report, but recent commits to make NPTL non-optional (and in particular enable it for x86-64 and i386) which will be in QEMU 1.6 should mean that the originally reported problem (of bash failing with "fork: Invalid argument") is fixed, and at least basic single-threaded x86 guest programs should work with linux-user. There may well be other issues still remaining for trying to run complex guest programs like Wine; if so please file fresh reports after retesting with 1.6.

[moonman (moonman-ca)]

Sorry to change the status. I'm not that familiar with Launchpad and was looking for a commit that fixes this bug.

[Peter Maydell (pmaydell)]

Commits aa004f5f9 to 24cb36a61c (13 in total) are the patchset that fix this.

[Justin Shafer (justinshafer)]

Wine works! =) Didn't know if you knew... no more old qemu.

You da man!

On Tue, Aug 6, 2013 at 3:33 AM, Peter Maydell <email address hidden>
wrote:

> Commits aa004f5f9 to 24cb36a61c (13 in total) are the patchset that fix
> this.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/739785
>
> Title:
> qemu-i386 user mode can't fork (bash: fork: Invalid argument)
>
> Status in QEMU:
> Fix Committed
> Status in “qemu” package in Debian:
> Confirmed
>
> Bug description:
> Good time of day everybody,
>
> I have been trying to make usermode qemu on ARM with plugapps
> (archlinux) with archlinux i386 chroot to work.
>
> 1. I installed arch linux in a virtuabox and created a chroot for it
> with mkarchroot. Transferred it to my pogo plug into /i386/
> 2. I comiled qemu-i386 static and put it into /i386/usr/bin/
> ./configure --static --disable-blobs --disable-system
> --target-list=i386-linux-user
> make
>
> 3. I also compiled linux kernel 2.6.38 with CONFIG_BINFMT_MISC=y and
> installed it.
> uname -a
> Linux Plugbox 2.6.38 #4 PREEMPT Fri Mar 18 22:19:10 CDT 2011 armv5tel
> Feroceon 88FR131 rev 1 (v5l) Marvell SheevaPlug Reference Board GNU/Linux
>
> 4. Added the following options into /etc/rc.local
> /sbin/modprobe binfmt_misc
> /bin/mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
> echo
> ':qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-i386:'
> >/proc/sys/fs/binfmt_misc/register
>
> 5. Also copied ld-linux.so.3 (actually ld-2.13.so because ld-
> linux.so.3 is a link to that file) from /lib/ to /i386/lib/
>
> 6.Now i chroot into /i386 and I get this:
> [root@Plugbox i386]# chroot .
> [II aI hnve ao n@P /]# pacman -Suy
> bash: fork: Invalid argument
>
> 7.I also downloaded linux-user-test-0.3 from qemu website and ran the
> test:
> [root@Plugbox linux-user-test-0.3]# make
> ./qemu-linux-user.sh
> [qemu-i386]
> ../qemu-0.14.0/i386-linux-user/qemu-i386 -L ./gnemul/qemu-i386 i386/ls
> -l dummyfile
> BUG IN DYNAMIC LINKER ld.so: dl-version.c: 210: _dl_check_map_versions:
> Assertion `needed != ((void *)0)' failed!
> make: *** [test] Error 127
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/739785/+subscriptions
>

[Justin Shafer (justinshafer)]

  ../qemu-0.14.0/i386-linux-user/qemu-i386 -L ./gnemul/qemu-i386 i386/ls -l
dummyfile

0.14.0??? I tried the latest qemu and it worked.. I forget the version..
1.XX something? I was able to run wine. It could also be your ld.so in
gnemul?

On Wed, Jun 15, 2016 at 7:41 AM, T. Huth <email address hidden> wrote:

> ** Changed in: qemu
> Status: Fix Committed => Fix Released
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/739785
>
> Title:
> qemu-i386 user mode can't fork (bash: fork: Invalid argument)
>
> Status in QEMU:
> Fix Released
> Status in qemu package in Debian:
> Fix Released
>
> Bug description:
> Good time of day everybody,
>
> I have been trying to make usermode qemu on ARM with plugapps
> (archlinux) with archlinux i386 chroot to work.
>
> 1. I installed arch linux in a virtuabox and created a chroot for it
> with mkarchroot. Transferred it to my pogo plug into /i386/
> 2. I comiled qemu-i386 static and put it into /i386/usr/bin/
> ./configure --static --disable-blobs --disable-system
> --target-list=i386-linux-user
> make
>
> 3. I also compiled linux kernel 2.6.38 with CONFIG_BINFMT_MISC=y and
> installed it.
> uname -a
> Linux Plugbox 2.6.38 #4 PREEMPT Fri Mar 18 22:19:10 CDT 2011 armv5tel
> Feroceon 88FR131 rev 1 (v5l) Marvell SheevaPlug Reference Board GNU/Linux
>
> 4. Added the following options into /etc/rc.local
> /sbin/modprobe binfmt_misc
> /bin/mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
> echo
> ':qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-i386:'
> >/proc/sys/fs/binfmt_misc/register
>
> 5. Also copied ld-linux.so.3 (actually ld-2.13.so because ld-
> linux.so.3 is a link to that file) from /lib/ to /i386/lib/
>
> 6.Now i chroot into /i386 and I get this:
> [root@Plugbox i386]# chroot .
> [II aI hnve ao n@P /]# pacman -Suy
> bash: fork: Invalid argument
>
> 7.I also downloaded linux-user-test-0.3 from qemu website and ran the
> test:
> [root@Plugbox linux-user-test-0.3]# make
> ./qemu-linux-user.sh
> [qemu-i386]
> ../qemu-0.14.0/i386-linux-user/qemu-i386 -L ./gnemul/qemu-i386 i386/ls
> -l dummyfile
> BUG IN DYNAMIC LINKER ld.so: dl-version.c: 210: _dl_check_map_versions:
> Assertion `needed != ((void *)0)' failed!
> make: *** [test] Error 127
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/739785/+subscriptions
>

--
Justin Shafer
Onsite Dental Systems
7704 Sagebrush Ct. S.
North Richland Hills, TX. 76182
[phone]