qemu-aarch64 cannot execute glibc

[Andreas Schwab]

$ aarch64-linux-user/qemu-aarch64 -version
qemu-aarch64 version 2.0.92, Copyright (c) 2003-2008 Fabrice Bellard
$ aarch64-linux-user/qemu-aarch64 -d in_asm /daten/build/build-root/home/abuild/rpmbuild/BUILD/glibc-2.19.90/cc-base/elf/ld-linux-aarch64.so.1
host mmap_min_addr=0x10000
guest_base 0x0
start end size prot
0000004000000000-000000400001e000 000000000001e000 r-x
000000400001e000-000000400002e000 0000000000010000 ---
000000400002e000-0000004000031000 0000000000003000 rw-
0000004000031000-0000004000032000 0000000000001000 ---
0000004000032000-0000004000832000 0000000000800000 rw-
start_brk 0x0000000000000000
end_code 0x000000400001dbe0
start_code 0x0000004000000000
start_data 0x000000400002eba8
end_data 0x0000004000030008
start_stack 0x0000004000830a10
brk 0x0000004000030170
entry 0x00000040000012c0
----------------
IN:
0x00000040000012c0: 910003e0 mov x0, sp
0x00000040000012c4: 94000d4f bl #+0x353c (addr 0x7fffb5bdad68)

----------------
IN: _dl_start
0x0000004000004800: d11243ff sub sp, sp, #0x490 (1168)
0x0000004000004804: a9ba7bfd stp x29, x30, [sp, #-96]!
0x0000004000004808: 910003fd mov x29, sp
0x000000400000480c: a9046bf9 stp x25, x26, [sp, #64]
0x0000004000004810: a90153f3 stp x19, x20, [sp, #16]
0x0000004000004814: a9025bf5 stp x21, x22, [sp, #32]
0x0000004000004818: a90363f7 stp x23, x24, [sp, #48]
0x000000400000481c: a90573fb stp x27, x28, [sp, #80]
0x0000004000004820: aa0003fa mov x26, x0
0x0000004000004824: d5033fdf isb
0x0000004000004828: d53be040 mrs x0, (unknown)

qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction
$ objdump -d /daten/build/build-root/home/abuild/rpmbuild/BUILD/glibc-2.19.90/cc-base/elf/ld-linux-aarch64.so.1 | grep ' 4828:'
    4828: d53be040 mrs x0, cntvct_el0

[Peter Maydell (pmaydell)]

On 18 July 2014 21:30, Andreas Schwab <email address hidden> wrote:
> qemu: uncaught target signal 4 (Illegal instruction) - core dumped
> Illegal instruction
> $ objdump -d /daten/build/build-root/home/abuild/rpmbuild/BUILD/glibc-2.19.90/cc-base/elf/ld-linux-aarch64.so.1 | grep ' 4828:'
> 4828: d53be040 mrs x0, cntvct_el0

That's awkward, it looks like glibc has started reading the
generic timer from userspace, and our implementation is
#ifndef CONFIG_USER_ONLY.

Do we have any current implementations of timers/counters
used by linux-user?

thanks
-- PMM

[Peter Maydell (pmaydell)]

Q: is this in a released glibc? The generic timers are strictly speaking optional, so this seems kind of bogus. Also looking at the glibc sources it never reads the counter frequency so what it is hoping to do with the timer values I have no idea.

[Peter Maydell (pmaydell)]

The kernel folks tell me that the generic timers being userspace accessible is only for the benefit of the kernel's VDSO, and they're not part of the kernel-to-userspace contract. This is therefore a bug in glibc, not QEMU, and we should get glibc fixed.

[Richard Henderson (rth)]

On 07/19/2014 12:54 AM, Peter Maydell wrote:
> On 18 July 2014 21:30, Andreas Schwab <email address hidden> wrote:
>> qemu: uncaught target signal 4 (Illegal instruction) - core dumped
>> Illegal instruction
>> $ objdump -d /daten/build/build-root/home/abuild/rpmbuild/BUILD/glibc-2.19.90/cc-base/elf/ld-linux-aarch64.so.1 | grep ' 4828:'
>> 4828: d53be040 mrs x0, cntvct_el0
>
> That's awkward, it looks like glibc has started reading the
> generic timer from userspace, and our implementation is
> #ifndef CONFIG_USER_ONLY.
>
> Do we have any current implementations of timers/counters
> used by linux-user?

Yes, several. We pass through to cpu_get_real_ticks.

r~

[Richard Henderson (rth)]

On 07/21/2014 12:46 AM, Peter Maydell wrote:
> Q: is this in a released glibc? The generic timers are strictly speaking
> optional, so this seems kind of bogus. Also looking at the glibc sources
> it never reads the counter frequency so what it is hoping to do with the
> timer values I have no idea.
>

It's trying to measure clock cycles required to perform the startup
relocations. E.g.

$ LD_DEBUG=statistics /bin/true
   4224:
   4224: runtime linker statistics:
   4224: total startup time in dynamic loader: 558705 clock cycles
   4224: time needed for relocation: 142093 clock cycles (25.4%)
   4224: number of relocations: 2
   4224: number of relocations from cache: 27
   4224: number of relative relocations: 0
   4224: time needed to load objects: 200809 clock cycles (35.9%)
   4224:
   4224: runtime linker statistics:
   4224: final number of relocations: 4
   4224: final number of relocations from cache: 27

r~

[Peter Maydell (pmaydell)]

On 21 July 2014 20:00, Richard Henderson <email address hidden> wrote:
> On 07/21/2014 12:46 AM, Peter Maydell wrote:
>> Q: is this in a released glibc? The generic timers are strictly speaking
>> optional, so this seems kind of bogus. Also looking at the glibc sources
>> it never reads the counter frequency so what it is hoping to do with the
>> timer values I have no idea.
>>
>
> It's trying to measure clock cycles required to perform the startup
> relocations.

That's a neat trick, given that the generic timers are not cycle
counters! They're a fixed frequency counter which is generally
unrelated and rather slower than the CPU frequency (and
which doesn't scale up and down with CPU frequency either).

-- PMM

[Richard Henderson (rth)]

On 07/21/2014 10:37 AM, Peter Maydell wrote:
>> It's trying to measure clock cycles required to perform the startup
>> relocations.
>
> That's a neat trick, given that the generic timers are not cycle
> counters! They're a fixed frequency counter which is generally
> unrelated and rather slower than the CPU frequency (and
> which doesn't scale up and down with CPU frequency either).

Even better.

r~

[Peter Maydell (pmaydell)]

On 22 July 2014 20:46, Richard Henderson <email address hidden> wrote:
> On 07/21/2014 10:37 AM, Peter Maydell wrote:
>>> It's trying to measure clock cycles required to perform the startup
>>> relocations.
>>
>> That's a neat trick, given that the generic timers are not cycle
>> counters! They're a fixed frequency counter which is generally
>> unrelated and rather slower than the CPU frequency (and
>> which doesn't scale up and down with CPU frequency either).
>
> Even better.

In any case the kernel guys say you can't guarantee they
exist unless you get them to define an ELF hwcap for
"timers exist and have a sane value in the 'what frequency
are they' register". So this is a glibc bug and I'm
not fixing QEMU...

-- PMM

[Riku Voipio (riku-voipio)]

On Tue, Jul 22, 2014 at 10:22:15PM -0000, Peter Maydell wrote:
> On 22 July 2014 20:46, Richard Henderson <email address hidden> wrote:
> > On 07/21/2014 10:37 AM, Peter Maydell wrote:
> >>> It's trying to measure clock cycles required to perform the startup
> >>> relocations.
> >>
> >> That's a neat trick, given that the generic timers are not cycle
> >> counters! They're a fixed frequency counter which is generally
> >> unrelated and rather slower than the CPU frequency (and
> >> which doesn't scale up and down with CPU frequency either).
> >
> > Even better.
>
> In any case the kernel guys say you can't guarantee they
> exist unless you get them to define an ELF hwcap for
> "timers exist and have a sane value in the 'what frequency
> are they' register". So this is a glibc bug and I'm
> not fixing QEMU...

It's used by openssl for rdtsc emulation as well:

http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/arm64cpuid.S#l17

Riku

> -- PMM
>
> Title:
> qemu-aarch64 cannot execute glibc
>
> Status in QEMU:
> New
>
> Bug description:
> $ aarch64-linux-user/qemu-aarch64 -version
> qemu-aarch64 version 2.0.92, Copyright (c) 2003-2008 Fabrice Bellard
> $ aarch64-linux-user/qemu-aarch64 -d in_asm /daten/build/build-root/home/abuild/rpmbuild/BUILD/glibc-2.19.90/cc-base/elf/ld-linux-aarch64.so.1
> host mmap_min_addr=0x10000
> guest_base 0x0
> start end size prot
> 0000004000000000-000000400001e000 000000000001e000 r-x
> 000000400001e000-000000400002e000 0000000000010000 ---
> 000000400002e000-0000004000031000 0000000000003000 rw-
> 0000004000031000-0000004000032000 0000000000001000 ---
> 0000004000032000-0000004000832000 0000000000800000 rw-
> start_brk 0x0000000000000000
> end_code 0x000000400001dbe0
> start_code 0x0000004000000000
> start_data 0x000000400002eba8
> end_data 0x0000004000030008
> start_stack 0x0000004000830a10
> brk 0x0000004000030170
> entry 0x00000040000012c0
> ----------------
> IN:
> 0x00000040000012c0: 910003e0 mov x0, sp
> 0x00000040000012c4: 94000d4f bl #+0x353c (addr 0x7fffb5bdad68)
>
> ----------------
> IN: _dl_start
> 0x0000004000004800: d11243ff sub sp, sp, #0x490 (1168)
> 0x0000004000004804: a9ba7bfd stp x29, x30, [sp, #-96]!
> 0x0000004000004808: 910003fd mov x29, sp
> 0x000000400000480c: a9046bf9 stp x25, x26, [sp, #64]
> 0x0000004000004810: a90153f3 stp x19, x20, [sp, #16]
> 0x0000004000004814: a9025bf5 stp x21, x22, [sp, #32]
> 0x0000004000004818: a90363f7 stp x23, x24, [sp, #48]
> 0x000000400000481c: a90573fb stp x27, x28, [sp, #80]
> 0x0000004000004820: aa0003fa mov x26, x0
> 0x0000004000004824: d5033fdf isb
> 0x0000004000004828: d53be040 mrs x0, (unknown)
>
> qemu: uncaught target signal 4 (Illegal instruction) - core dumped
> Illegal instruction
> $ objdump -d /daten/build/build-root/home/abuild/rpmbuild/BUILD/glibc-2.19.90/cc-base/elf/ld-linux-aarch64.so.1 | grep ' 4828:'
> 4828: d53be040 mrs x0, cntvct_el0
>
> To manage notifications about this bug go to:
>...

[Ard Biesheuvel (ard-biesheuvel)]

On 25 July 2014 10:54, Riku Voipio <email address hidden> wrote:
> On Tue, Jul 22, 2014 at 10:22:15PM -0000, Peter Maydell wrote:
>> On 22 July 2014 20:46, Richard Henderson <email address hidden> wrote:
>> > On 07/21/2014 10:37 AM, Peter Maydell wrote:
>> >>> It's trying to measure clock cycles required to perform the startup
>> >>> relocations.
>> >>
>> >> That's a neat trick, given that the generic timers are not cycle
>> >> counters! They're a fixed frequency counter which is generally
>> >> unrelated and rather slower than the CPU frequency (and
>> >> which doesn't scale up and down with CPU frequency either).
>> >
>> > Even better.
>>
>> In any case the kernel guys say you can't guarantee they
>> exist unless you get them to define an ELF hwcap for
>> "timers exist and have a sane value in the 'what frequency
>> are they' register". So this is a glibc bug and I'm
>> not fixing QEMU...
>
> It's used by openssl for rdtsc emulation as well:
>
> http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/arm64cpuid.S#l17
>

It is well known to me and to the upstream author that the timer count
is meaningless without also reading the frequency.
However, the piece of code you refer to is only used for internal
development by the upstream author, so I don't think we should care
about it.
(The snippet in question is executed with a SIGILL trap set as a kind
of poor man's elf_hwcap)

--
Ard.

>> -- PMM
>>
>> Title:
>> qemu-aarch64 cannot execute glibc
>>
>> Status in QEMU:
>> New
>>
>> Bug description:
>> $ aarch64-linux-user/qemu-aarch64 -version
>> qemu-aarch64 version 2.0.92, Copyright (c) 2003-2008 Fabrice Bellard
>> $ aarch64-linux-user/qemu-aarch64 -d in_asm /daten/build/build-root/home/abuild/rpmbuild/BUILD/glibc-2.19.90/cc-base/elf/ld-linux-aarch64.so.1
>> host mmap_min_addr=0x10000
>> guest_base 0x0
>> start end size prot
>> 0000004000000000-000000400001e000 000000000001e000 r-x
>> 000000400001e000-000000400002e000 0000000000010000 ---
>> 000000400002e000-0000004000031000 0000000000003000 rw-
>> 0000004000031000-0000004000032000 0000000000001000 ---
>> 0000004000032000-0000004000832000 0000000000800000 rw-
>> start_brk 0x0000000000000000
>> end_code 0x000000400001dbe0
>> start_code 0x0000004000000000
>> start_data 0x000000400002eba8
>> end_data 0x0000004000030008
>> start_stack 0x0000004000830a10
>> brk 0x0000004000030170
>> entry 0x00000040000012c0
>> ----------------
>> IN:
>> 0x00000040000012c0: 910003e0 mov x0, sp
>> 0x00000040000012c4: 94000d4f bl #+0x353c (addr 0x7fffb5bdad68)
>>
>> ----------------
>> IN: _dl_start
>> 0x0000004000004800: d11243ff sub sp, sp, #0x490 (1168)
>> 0x0000004000004804: a9ba7bfd stp x29, x30, [sp, #-96]!
>> 0x0000004000004808: 910003fd mov x29, sp
>> 0x000000400000480c: a9046bf9 stp x25, x26, [sp, #64]
>> 0x0000004000004810: a90153f3 stp x19, x20, [sp, #16]
>> 0x0000004000004814: a9025bf5 stp x21, x22, [sp, #32]
>> 0x0000004000004818: a90363f7 stp x23, x24, [sp, #48]
>> 0x000000400000481c: a90573fb stp x27, x28, [sp, #8...

[Peter Maydell (pmaydell)]

To summarise the situation here:

 * in ARMv8 the Generic Timers are mandatory at the architectural/hardware level
 * however the kernel folks do not want to guarantee that they are always exposed to userspace (they want the flexibility in future to disable the userspace access for possible errata workarounds on future boards/CPUs)
 * the glibc change has been reverted upstream: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=33ef2f0c763b51e1df7896d7d39d585824558c75

So I'm closing this bug because QEMU's behaviour currently is fine. If the kernel folk define a hwcap for timer access we can consider both exposing the generic timers and setting the hwcap then. (Alternatively if we ever get round to defining a VDSO for QEMU linux-user mode we might want the timer access for that, as the kernel does.)