Libraries compiled without Shadow Stack support

[Marcos Alano]

I tried to execute VLC when I forcefully enabled Shadow Stack on my system using the environment variable GLIBC_TUNABLES=glibc.cpu.hwcaps=SHSTK and it didn't run correctly:

➜ GLIBC_TUNABLES=glibc.cpu.hwcaps=SHSTK vlc -v
VLC media player 3.0.20 Vetinari (revision 3.0.20-0-g6f0d0ab126b)
[000060bf9270b5e0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[000060bf927a3460] main playlist: playlist is empty
☸ rancher-desktop in packages/fwupd/1.9.20 on ☁️ (us-east-1) on ☁️ <email address hidden> took 3s
➜ GLIBC_TUNABLES=glibc.cpu.hwcaps=SHSTK vlc -v
VLC media player 3.0.20 Vetinari (revision 3.0.20-0-g6f0d0ab126b)
[00006399376a4e90] main audio output warning: cannot load module `/usr/lib/x86_64-linux-gnu/vlc/plugins/audio_output/libpulse_plugin.so' (/lib/x86_64-linux-gnu/libmpg123.so.0: rebuild shared object with SHSTK support enabled)
[00006399375e95e0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[00006399377027e0] main interface warning: cannot load module `/usr/lib/x86_64-linux-gnu/vlc/plugins/gui/libqt_plugin.so' (/lib/x86_64-linux-gnu/libQt5Core.so.5: rebuild shared object with SHSTK support enabled)
[00007d14880245f0] main generic warning: cannot load module `/usr/lib/x86_64-linux-gnu/vlc/plugins/gui/libqt_plugin.so' (/lib/x86_64-linux-gnu/libQt5Core.so.5: rebuild shared object with SHSTK support enabled)
[00006399377027e0] skins2 interface error: cannot instantiate dialogs provider
[0000639937681460] main playlist: playlist is empty
[00006399377027e0] [cli] lua interface: Listening on host "*console".
VLC media player 3.0.20 Vetinari
Command Line Interface initialized. Type `help' for help.
>

If I check the library, it doesn't show Shadow Stack enabled. The command:
readelf -a /lib/x86_64-linux-gnu/libmpg123.so.0|grep STK
Returns nothing.
As control group, I tried with libssl3:
➜ readelf -a /lib/x86_64-linux-gnu/libssl.so.3 |grep STK
      Properties: x86 feature: IBT, SHSTK
The same problem happened with:
/lib/x86_64-linux-gnu/libQt5Core.so.5

[Thomas Orgis (thomas-forum)]

Am Mon, 20 May 2024 23:51:15 -0000
schrieb Marcos Alano <email address hidden>:

> [00006399376a4e90] main audio output warning: cannot load module `/usr/lib/x86_64-linux-gnu/vlc/plugins/audio_output/libpulse_plugin.so' (/lib/x86_64-linux-gnu/libmpg123.so.0: rebuild shared object with SHSTK support enabled)

Regarding libmpg123, you either need to disable assembly optimizations
(build with generic decoders only), I presume, or someone provide a
patch that adds SHSTK to them. I don't know which implementation of
shadow stacks glibc/gcc is using in that setup. I've read up on the
concept and so far only figured that this is part of a spiral that
complicates ABI and makes providing assembly-optimized functions ever
harder. This would be fine if compilers finally would be smart enough
to evade the need to do so. Last time I checked, hand-tuned AVX
decoding was still a lot more efficient.

We already handle IBT, I think, with indirect jumps landing only in C
wrapper functions. I wonder if we could also limit the shadow stack
impact to those with some compiler/linker flags. The assembly routines
are rather strict math, many years old now without much of attack
surface. All parsing of input is before them in C. They just do lots of
multiplication/addition.

One might try to write a set of optimizations using intrinsics for
modern CPUs that then also get the treatment of shadow stacks or the
next shiny security measure. Porting the AVX code to GCC (and/or other)
inline ASM might also work for some platforms.

(Still, I am wondering why pulseaudio output should need MPEG decoding.)

Alrighty then,

Thomas