librte-net22 / librte_net.so.22 SIGILL in rte_srand()

[Garrett Comeaux]

I think this bug has been reported against gqrx several times, but one can duplicate the issue by installing `uhd-host` on a system without sse4_2 extensions and executing `uhd_usrp_probe`.

The `pcmpgtq` instruction requires sse4_2, and my system only has sse4_1.
$ dpkg -S /usr/lib/x86_64-linux-gnu/librte_net.so.22
librte-net22:amd64: /usr/lib/x86_64-linux-gnu/librte_net.so.22

+

See https://github.com/gqrx-sdr/gqrx/issues/1180

Thanks for considering.

```
(gdb) run
Starting program: /usr/bin/uhd_usrp_probe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0x00007ffff6f1e684 in __rte_rand_lfsr258_gen_seed (min_value=<optimized out>, seed=<optimized out>) at ../lib/eal/common/rte_random.c:60
Download failed: Invalid argument. Continuing without source file ./obj-x86_64-linux-gnu/../lib/eal/common/rte_random.c.
60 ../lib/eal/common/rte_random.c: No such file or directory.
(gdb) bt
#0 0x00007ffff6f1e684 in __rte_rand_lfsr258_gen_seed (min_value=<optimized out>, seed=<optimized out>)
    at ../lib/eal/common/rte_random.c:60
#1 __rte_srand_lfsr258 (state=<optimized out>, seed=<optimized out>) at ../lib/eal/common/rte_random.c:72
#2 rte_srand (seed=11813869339904743830) at ../lib/eal/common/rte_random.c:86
#3 0x00007ffff6f128cc in _sub_I_65535_0.0 () from /lib/x86_64-linux-gnu/librte_eal.so.22
#4 0x00007ffff7fcbfbe in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffdfc8, env=env@entry=0x7fffffffdfd8)
    at ./elf/dl-init.c:70
#5 0x00007ffff7fcc0a8 in call_init (env=0x7fffffffdfd8, argv=0x7fffffffdfc8, argc=1, l=<optimized out>) at ./elf/dl-init.c:33
#6 _dl_init (main_map=0x7ffff7ffe2e0, argc=1, argv=0x7fffffffdfc8, env=0x7fffffffdfd8) at ./elf/dl-init.c:117
#7 0x00007ffff7fe48b0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#8 0x0000000000000001 in ?? ()
#9 0x00007fffffffe307 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb) disassemble __rte_rand_lfsr258_gen_seed
Dump of assembler code for function rte_srand:
   0x00007ffff6f1e560 <+0>: endbr64
   0x00007ffff6f1e564 <+4>: movq %rdi,%xmm2
   0x00007ffff6f1e569 <+9>: movdqa 0x1b4bf(%rip),%xmm3 # 0x7ffff6f39a30
   0x00007ffff6f1e571 <+17>: punpcklqdq %xmm2,%xmm2
   0x00007ffff6f1e575 <+21>: lea 0x48684(%rip),%rax # 0x7ffff6f66c00 <rand_states>
   0x00007ffff6f1e57c <+28>: movaps %xmm2,-0x18(%rsp)
   0x00007ffff6f1e581 <+33>: movdqa 0x1b486(%rip),%xmm13 # 0x7ffff6f39a10
   0x00007ffff6f1e58a <+42>: movdqa 0x1b4ae(%rip),%xmm2 # 0x7ffff6f39a40
   0x00007ffff6f1e592 <+50>: lea 0x2000(%rax),%rdx
   0x00007ffff6f1e599 <+57>: movdqa 0x1b4be(%rip),%xmm14 # 0x7ffff6f39a60
   0x00007ffff6f1e5a2 <+66>: nopw 0x0(%rax,%rax,1)
   0x00007ffff6f1e5a8 <+72>: movdqa %xmm13,%xmm1
   0x00007ffff6f1e5ad <+77>: movdqa -0x18(%rsp),%xmm7
   0x00007ffff6f1e5b3 <+83>: paddd 0x1b464(%rip),%xmm13 # 0x7ffff6f39a20
   0x00007ffff6f1e5bc <+92>: add $0x100,%rax
   0x00007ffff6f1e5c2 <+98>: movdqa 0x1b485(%rip),%xmm11 # 0x7ffff6f39a50
   0x00007ffff6f1e5cb <+107>: pmovzxdq %xmm1,%xmm4
   0x00007ffff6f1e5d0 <+112>: psrldq $0x8,%xmm1
   0x00007ffff6f1e5d5 <+117>: pmovzxdq %xmm1,%xmm1
   0x00007ffff6f1e5da <+122>: paddq %xmm7,%xmm4
   0x00007ffff6f1e5de <+126>: movdqa 0x1b489(%rip),%xmm10 # 0x7ffff6f39a70
   0x00007ffff6f1e5e7 <+135>: paddq %xmm7,%xmm1
   0x00007ffff6f1e5eb <+139>: movdqa %xmm4,%xmm0
   0x00007ffff6f1e5ef <+143>: movdqa 0x1b478(%rip),%xmm9 # 0x7ffff6f39a70
   0x00007ffff6f1e5f8 <+152>: movdqa %xmm1,%xmm5
   0x00007ffff6f1e5fc <+156>: psrlq $0x20,%xmm0
   0x00007ffff6f1e601 <+161>: shufps $0x88,%xmm1,%xmm4
   0x00007ffff6f1e605 <+165>: movdqa 0x1b472(%rip),%xmm8 # 0x7ffff6f39a80
   0x00007ffff6f1e60e <+174>: psrlq $0x20,%xmm5
   0x00007ffff6f1e613 <+179>: movdqa 0x1b465(%rip),%xmm7 # 0x7ffff6f39a80
   0x00007ffff6f1e61b <+187>: shufps $0x88,%xmm5,%xmm0
   0x00007ffff6f1e61f <+191>: pxor %xmm4,%xmm0
   0x00007ffff6f1e623 <+195>: pmulld %xmm3,%xmm0
   0x00007ffff6f1e628 <+200>: paddd %xmm2,%xmm0
   0x00007ffff6f1e62c <+204>: movdqa %xmm0,%xmm1
   0x00007ffff6f1e630 <+208>: pmovzxdq %xmm0,%xmm4
   0x00007ffff6f1e635 <+213>: pmulld %xmm3,%xmm1
   0x00007ffff6f1e63a <+218>: psrldq $0x8,%xmm0
   0x00007ffff6f1e63f <+223>: pmovzxdq %xmm0,%xmm0
   0x00007ffff6f1e644 <+228>: paddd %xmm2,%xmm1
--Type <RET> for more, q to quit, c to continue without paging--
   0x00007ffff6f1e648 <+232>: pmovzxdq %xmm1,%xmm5
   0x00007ffff6f1e64d <+237>: psllq $0x20,%xmm5
   0x00007ffff6f1e652 <+242>: por %xmm4,%xmm5
   0x00007ffff6f1e656 <+246>: movdqa %xmm1,%xmm4
   0x00007ffff6f1e65a <+250>: paddq %xmm5,%xmm11
   0x00007ffff6f1e65f <+255>: pmulld %xmm3,%xmm1
   0x00007ffff6f1e664 <+260>: psrldq $0x8,%xmm4
   0x00007ffff6f1e669 <+265>: pmovzxdq %xmm4,%xmm4
   0x00007ffff6f1e66e <+270>: psllq $0x20,%xmm4
   0x00007ffff6f1e673 <+275>: paddd %xmm2,%xmm1
   0x00007ffff6f1e677 <+279>: por %xmm0,%xmm4
   0x00007ffff6f1e67b <+283>: movdqa %xmm5,%xmm0
   0x00007ffff6f1e67f <+287>: psubq %xmm14,%xmm0
=> 0x00007ffff6f1e684 <+292>: pcmpgtq 0x1b423(%rip),%xmm0 # 0x7ffff6f39ab0
   0x00007ffff6f1e68d <+301>: pblendvb %xmm0,%xmm5,%xmm11
   0x00007ffff6f1e693 <+307>: movdqa %xmm4,%xmm0
   0x00007ffff6f1e697 <+311>: movdqa 0x1b3b1(%rip),%xmm5 # 0x7ffff6f39a50
   0x00007ffff6f1e69f <+319>: movq %xmm11,-0x100(%rax)
   0x00007ffff6f1e6a8 <+328>: psubq %xmm14,%xmm0
   0x00007ffff6f1e6ad <+333>: pextrq $0x1,%xmm11,-0xc0(%rax)
   0x00007ffff6f1e6b8 <+344>: pcmpgtq 0x1b3ef(%rip),%xmm0 # 0x7ffff6f39ab0
   0x00007ffff6f1e6c1 <+353>: paddq %xmm4,%xmm5
```

ProblemType: Crash
DistroRelease: Ubuntu 22.10
Package: gqrx-sdr 2.15.9-1
ProcVersionSignature: Ubuntu 5.19.0-31.32-generic 5.19.17
Uname: Linux 5.19.0-31-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.23.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: i3
Date: Mon Mar 6 16:25:45 2023
ExecutablePath: /usr/bin/gqrx
ExecutableTimestamp: 1661386628
InstallationDate: Installed on 2022-01-05 (425 days ago)
InstallationMedia: Ubuntu 20.04.3 LTS "Focal Fossa" - Release amd64 (20210819)
JournalErrors: -- No entries --
ProcCmdline: gqrx
ProcCwd: /home/garrett/Code/gqrx/build
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 TERM=rxvt-unicode-256color
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, user)
RebootRequiredPkgs: Error: path contained symlinks.
Signal: 4
SourcePackage: gqrx-sdr
StacktraceTop:
 rte_srand () from /lib/x86_64-linux-gnu/librte_eal.so.22
 ?? () from /lib/x86_64-linux-gnu/librte_eal.so.22
 call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fff2573be68, env=env@entry=0x7fff2573be78) at ./elf/dl-init.c:70
 call_init (env=0x7fff2573be78, argv=0x7fff2573be68, argc=1, l=<optimized out>) at ./elf/dl-init.c:33
 _dl_init (main_map=0x7f7f8bd532e0, argc=1, argv=0x7fff2573be68, env=0x7fff2573be78) at ./elf/dl-init.c:117
Title: gqrx crashed with SIGILL in rte_srand()
UpgradeStatus: Upgraded to kinetic on 2022-12-28 (68 days ago)
UserGroups: adm cdrom dialout dip disk docker libvirt lpadmin lxd plugdev sambashare sudo wireshark
mtime.conffile..etc.apport.crashdb.conf: 2022-08-24T09:03:25.873541
separator:

[Apport retracing service (apport)]

Thank you for taking the time to report this crash and helping to make this software better. This particular crash has already been reported and is a duplicate of bug #2009540, so is being marked as such. Please look at the other bug report to see if there is any missing information that you can provide, or to see if there is a workaround for the bug. Additionally, any further discussion regarding the bug should occur in the other report. Please continue to report any other bugs you may find.

[Christian Ehrhardt  (paelzer)]

Hi Garret,
this is a bug I was waiting for a long time, I thought we were over it by time passing by and HW getting newer.

Upstream dpdk decided years ago that SSE4.2 (and crc32 on arm for that matter) is the lowest common denominator. That is higher than the usual baseline of Ubuntu, but there isn't much we can do about it.

You can track [1] back to almost the eternity of the DPDK project.

This isn't a regression, nor a bug in the classic meaning.

If you have a strong use case for HW that won't have this feature in the future (I mean my almost dying 8 year old server has this, so it is not new), but makes sense for DPDK to run on you might bring it up with the DPDK community [2] to consider making this a non-hard requirement (many other things are run-time optimizations).

From just Ubuntu's POV I'm not sure what we could do right now :-/

[1]: https://github.com/DPDK/dpdk/blame/main/config/x86/meson.build#L28
[2]: https://bugs.dpdk.org/

[Garrett Comeaux (gmcomeaux)]

No, I completely understand. I was trying to think of a graceful way that
the SSE4.2 extensions could be checked for and an error reported instead of
an illegal instruction. I guess that's about the best you can do. Anyway,
thanks for the explanation.

On Wed, Mar 8, 2023, 06:40 Christian Ehrhardt  <email address hidden>
wrote:

> *** This bug is a duplicate of bug 2009540 ***
> https://bugs.launchpad.net/bugs/2009540
>
> Hi Garret,
> this is a bug I was waiting for a long time, I thought we were over it by
> time passing by and HW getting newer.
>
> Upstream dpdk decided years ago that SSE4.2 (and crc32 on arm for that
> matter) is the lowest common denominator. That is higher than the usual
> baseline of Ubuntu, but there isn't much we can do about it.
>
> You can track [1] back to almost the eternity of the DPDK project.
>
> This isn't a regression, nor a bug in the classic meaning.
>
> If you have a strong use case for HW that won't have this feature in the
> future (I mean my almost dying 8 year old server has this, so it is not
> new), but makes sense for DPDK to run on you might bring it up with the
> DPDK community [2] to consider making this a non-hard requirement (many
> other things are run-time optimizations).
>
> From just Ubuntu's POV I'm not sure what we could do right now :-/
>
> [1]: https://github.com/DPDK/dpdk/blame/main/config/x86/meson.build#L28
> [2]: https://bugs.dpdk.org/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2009635
>
> Title:
> librte-net22 / librte_net.so.22 SIGILL in rte_srand()
>
> Status in dpdk package in Ubuntu:
> New
>
> Bug description:
> I think this bug has been reported against gqrx several times, but one
> can duplicate the issue by installing `uhd-host` on a system without
> sse4_2 extensions and executing `uhd_usrp_probe`.
>
> The `pcmpgtq` instruction requires sse4_2, and my system only has sse4_1.
> $ dpkg -S /usr/lib/x86_64-linux-gnu/librte_net.so.22
> librte-net22:amd64: /usr/lib/x86_64-linux-gnu/librte_net.so.22
>
> +
>
> See https://github.com/gqrx-sdr/gqrx/issues/1180
>
> Thanks for considering.
>
>
> ```
> (gdb) run
>
> Starting program: /usr/bin/uhd_usrp_probe
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library
> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
> Program received signal SIGILL, Illegal instruction.
> 0x00007ffff6f1e684 in __rte_rand_lfsr258_gen_seed (min_value=<optimized
> out>, seed=<optimized out>) at ../lib/eal/common/rte_random.c:60
> Download failed: Invalid argument. Continuing without source file
> ./obj-x86_64-linux-gnu/../lib/eal/common/rte_random.c.
> 60 ../lib/eal/common/rte_random.c: No such file or directory.
> (gdb) bt
> #0 0x00007ffff6f1e684 in __rte_rand_lfsr258_gen_seed
> (min_value=<optimized out>, seed=<optimized out>)
> at ../lib/eal/common/rte_random.c:60
> #1 __rte_srand_lfsr258 (state=<optimized out>, seed=<optimized out>) at
> ../lib/eal/common/rte_random.c:72
> #2 rte_srand (seed=11813869339904743830) at
> ../lib/eal/common/rte_random.c:86
> #3 0x00007ffff6f128cc in _...

[Christian Ehrhardt  (paelzer)]

Since the discussion is further here, let us revert the duplication and add a bug task here (even though we are not yet sure what to do about it).

From the other bug report:
I've got an older machine, it's a Dell Precision T5400 with Intel(R) Xeon(R) CPU E5420 @ 2.50GH. The latest SSE extensions it has are sse4_1, according to /proc/cpuinfo.

It looks like librte-eal22 was built to require sse4_2, and illegal instruction, pcmpgtq, was introduced in sse4_2.

So, can we have librte-eal22 built without requiring those extensions, or should I just update my hardware? :-)

[Christian Ehrhardt  (paelzer)]

> I was trying to think of a graceful way that the SSE4.2 extensions could be checked for and an error reported instead of an illegal instruction.

That is possible, I'd wish that could be a single check at just one lib initializing.
How is the full lddtree look like of your example?

Usually even if you link to a particular sub-lib of dpdk that will in turn link to librte_eal.so eventually - so it could be done in just one place.

In fact I think that was already done for cpu features, e.g. the former (21.11) SIGILL on arm for CRC32 nowadays looks like:
  HASH: Unsupported CRC32 algorithm requested using CRC32_SW
  ERROR: This system does not support "CRC32".
  Please check that RTE_MACHINE is set correctly.
  EAL: FATAL: unsupported cpu type.

Could you try Ubuntu Lunar (23.04) which has DPDK 22.11.?
That will still not work, but I wonder if x86 - and sse4.2 on that arch - got a similar "now graceful" check.

> I guess that's about the best you can do. Anyway, thanks for the explanation.

yes, Unless we can convince for full runtime-detection sse4.2

> I've got an older machine ...

That is the reason why I think it might not be addressed at high-prio by anyone.
It was uncommon years ago to lack this. And it is even more uncommon nowadays and will be more uncommon in the future.

[Christian Ehrhardt  (paelzer)]

Hmm, I rechecked my logs in detail it still is SIGinvalid but with the message above.
Not perfect, but worth to check on your system before reporting upstream.

[Christian Ehrhardt  (paelzer)]

And if you do test 22.11 maybe try to give a full backtrace, that might help to consider if it could be opted out (imagine your use UHD but do not care for dpdk, then there should be a way to skip). Therefore it depends if it is something like a constructor on load or an actual use of dpdk.

[Garrett Comeaux (gmcomeaux)]

I ran an ubuntu:lunar docker container and install DPDK 22.11. It seems that librte_eal.so is loaded. I've included the full ldd output below. I still don't get a graceful check, though.
```
root@1bee22d20ca0:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Lunar Lobster (development branch)
Release: 23.04
Codename: lunar

root@1bee22d20ca0:/# apt info dpdk
Package: dpdk
Version: 22.11.1-2ubuntu1
Priority: optional
Section: devel
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian DPDK Maintainers <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 106 kB
Depends: lsb-base (>= 3.2-14), pci.ids | hwdata, pciutils, procps, python3:any
Recommends: librte-mempool23, librte-mempool-ring23, librte-net-af-packet23, librte-net-bond23, librte-net-e1000-23, librte-net-fm10k23, librte-net-i40e23, librte-net-ixgbe23, librte-net-mlx4-23, librte-net-mlx5-23, librte-net-netvsc23, librte-net-pcap23, librte-net-tap23, librte-net-thunderx23, librte-net-vdev-netvsc23, librte-net-vhost23, librte-net-virtio23, librte-net-vmxnet3-23, python3-pyelftools
Suggests: dpdk-doc, dpdk-kmods-dkms, librte-meta-baseband, librte-meta-bus, librte-meta-common, librte-meta-compress, librte-meta-crypto, librte-meta-dma, librte-meta-event, librte-meta-mempool, librte-meta-net, librte-meta-raw, librte-meta-allpmds
Breaks: dpdk-dev (<< 18.11-4~)
Replaces: dpdk-dev (<< 18.11)
Homepage: https://www.dpdk.org
Download-Size: 29.5 kB
APT-Sources: http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
Description: Data Plane Development Kit (runtime)

root@1bee22d20ca0:/# uname -a
Linux 1bee22d20ca0 5.19.0-31-generic #32-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 20 15:20:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

root@1bee22d20ca0:/# uhd_usrp_probe
Illegal instruction (core dumped)

(gdb) bt
#0 0x00007f4b2d3a3374 in rte_srand () from /lib/x86_64-linux-gnu/librte_eal.so.23
#1 0x00007f4b2d3967ec in ?? () from /lib/x86_64-linux-gnu/librte_eal.so.23
#2 0x00007f4b2e5d1fbe in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7ffeabf5b488, env=env@entry=0x7ffeabf5b498)
    at ./elf/dl-init.c:70
#3 0x00007f4b2e5d20a8 in call_init (env=0x7ffeabf5b498, argv=0x7ffeabf5b488, argc=1, l=<optimized out>) at ./elf/dl-init.c:33
#4 _dl_init (main_map=0x7f4b2e6042e0, argc=1, argv=0x7ffeabf5b488, env=0x7ffeabf5b498) at ./elf/dl-init.c:117
#5 0x00007f4b2e5ea8b0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#6 0x0000000000000001 in ?? ()
#7 0x00007ffeabf5c844 in ?? ()
#8 0x0000000000000000 in ?? ()

root@1bee22d20ca0:/# ldd /usr/bin/uhd_usrp_probe
 linux-vdso.so.1 (0x00007ffc2a3b7000)
 libuhd.so.4.3.0 => /lib/x86_64-linux-gnu/libuhd.so.4.3.0 (0x00007f44bcd03000)
 libboost_program_options.so.1.74.0 => /lib/x86_64-linux-gnu/libboost_program_options.so.1.74.0 (0x00007f44bccbf000)
 libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f44bca55000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f44bca31000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f44bc82c000)
 libboost_chrono.so.1.74.0 => /...

[Christian Ehrhardt  (paelzer)]

Thanks for the try.
I've asked upstream [1] what they'd think of a more graceful lib loading code.
Let us see what they answer and if they agree to do something about it if that is backportable to existing Ubuntu releases.

[1]: http://mails.dpdk.org/archives/dev/2023-March/265921.html