pollen crashes in containers

[Christian Ehrhardt ]

In jammy or focal containers I found pollen to be crashing very very early.

Some actions to debug early

root@j:~# pollen
Segmentation fault

In gdb you only get:
"During startup program terminated with signal SIGSEGV, Segmentation fault."

Acting on some "get early issues" gives me this:

(gdb) catch throw
Catchpoint 1 (throw)
(gdb) starti
Starting program: /usr/bin/pollen
During startup program terminated with signal SIGSEGV, Segmentation fault.

Still not enough, catch the entry point:

(gdb) info files
Symbols from "/usr/bin/pollen".
Local exec file:
 `/usr/bin/pollen', file type elf64-x86-64.
 Entry point: 0x46c9c0
 0x0000000000401000 - 0x000000000065bb33 is .text
 0x000000000065bb40 - 0x000000000065bd60 is .plt
 0x000000000065c000 - 0x0000000000733103 is .rodata
 0x0000000000733a80 - 0x0000000000733e10 is .dynsym
 0x0000000000733108 - 0x0000000000733120 is .rela
 0x0000000000733120 - 0x0000000000733438 is .rela.plt
 0x0000000000733440 - 0x000000000073348c is .gnu.version
 0x00000000007334a0 - 0x0000000000733520 is .gnu.version_r
 0x0000000000733520 - 0x00000000007335dc is .hash
 0x0000000000733820 - 0x0000000000733a71 is .dynstr
 0x0000000000733e20 - 0x000000000073550c is .typelink
 0x0000000000735510 - 0x0000000000735d58 is .itablink
 0x0000000000735d58 - 0x0000000000735d58 is .gosymtab
 0x0000000000735d60 - 0x000000000085f25e is .gopclntab
 0x0000000000860000 - 0x0000000000860020 is .go.buildinfo
 0x0000000000860020 - 0x0000000000860140 is .got.plt
 0x0000000000860140 - 0x0000000000860270 is .dynamic
 0x0000000000860270 - 0x0000000000860278 is .got
 0x0000000000860280 - 0x00000000008968a0 is .noptrdata
 0x00000000008968a0 - 0x00000000008a20b0 is .data
 0x00000000008a20c0 - 0x00000000008d3a90 is .bss
 0x00000000008d3aa0 - 0x00000000008d7648 is .noptrbss
 0x0000000000000000 - 0x0000000000000008 is .tbss
 0x0000000000400fe4 - 0x0000000000401000 is .interp
 0x0000000000400f80 - 0x0000000000400fe4 is .note.go.buildid
(gdb) break *0x46c9c0
Breakpoint 3 at 0x46c9c0: file /usr/lib/go-1.15/src/runtime/rt0_linux_amd64.s, line 8.
(gdb) r
Starting program: /usr/bin/pollen
During startup program terminated with signal SIGSEGV, Segmentation fault.

Really ?!?, debug loading ...

(gdb) export LD_DEBUG=all
Undefined command: "export". Try "help".
(gdb) set environment LD_DEBUG=all
(gdb) r
Starting program: /usr/bin/pollen
     21332: symbol=__vdso_clock_gettime; lookup in file=linux-vdso.so.1 [0]
     21332: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_clock_gettime' [LINUX_2.6]
...
     21332:
     21332: calling init: /lib64/ld-linux-x86-64.so.2
     21332:
     21332:
     21332: calling init: /lib/x86_64-linux-gnu/libc.so.6
     21332:
     21332:
     21332: initialize program: /bin/sh
     21332:
     21332: symbol=_dl_audit_preinit; lookup in file=/bin/sh [0]
     21332: symbol=_dl_audit_preinit; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
     21332: symbol=_dl_audit_preinit; lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     21332: binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `_dl_audit_preinit' [GLIBC_PRIVATE]
     21332:
     21332: transferring control: /bin/sh
     21332:
During startup program terminated with signal SIGSEGV, Segmentation fault.

I might miss the cause in that huge output, but I didn't see something there either.

Rebuilding it locally (just make and also via debuild) creates a binary that works.
Also it does work fine in a Jammy and Focal VM,
only in LXD containers I see this breakage.

[Christian Ehrhardt  (paelzer)]

Full LD debug

(gdb) export LD_DEBUG=all
Undefined command: "export". Try "help".
(gdb) set environment LD_DEBUG=all
(gdb) r
Starting program: /usr/bin/pollen
     21332: symbol=__vdso_clock_gettime; lookup in file=linux-vdso.so.1 [0]
     21332: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_clock_gettime' [LINUX_2.6]
     21332: symbol=__vdso_gettimeofday; lookup in file=linux-vdso.so.1 [0]
     21332: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_gettimeofday' [LINUX_2.6]
     21332: symbol=__vdso_time; lookup in file=linux-vdso.so.1 [0]
     21332: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_time' [LINUX_2.6]
     21332: symbol=__vdso_getcpu; lookup in file=linux-vdso.so.1 [0]
     21332: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_getcpu' [LINUX_2.6]
     21332: symbol=__vdso_clock_getres; lookup in file=linux-vdso.so.1 [0]
     21332: binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_clock_getres' [LINUX_2.6]
     21332:
     21332: file=libc.so.6 [0]; needed by /bin/sh [0]
     21332: find library=libc.so.6 [0]; searching
     21332: search cache=/etc/ld.so.cache
     21332: trying file=/lib/x86_64-linux-gnu/libc.so.6
     21332:
     21332: file=libc.so.6 [0]; generating link map
     21332: dynamic: 0x00007ffff7fa1bc0 base: 0x00007ffff7d89000 size: 0x0000000000227e50
     21332: entry: 0x00007ffff7db2f50 phdr: 0x00007ffff7d89040 phnum: 14
     21332:
     21332: checking for version `GLIBC_2.3' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.11' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.3.4' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.14' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.33' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.4' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.34' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.2.5' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /bin/sh [0]
     21332: checking for version `GLIBC_2.2.5' in file /lib64/ld-linux-x86-64.so.2 [0] required by file /lib/x86_64-linux-gnu/libc.so.6 [0]
     21332: checking for version `GLIBC_2.3' in file /lib64/ld-linux-x86-64.so.2 [0] required by file /lib/x86_64-linux-gnu/libc.so.6 [0]
     21332: checking for version `GLIBC_PRIVATE' in file /lib64/ld-linux-x86-64.so.2 [0] required by file /lib/x86_64-linux-gnu/libc.so.6 [0]
     21332:
     21332: Initial object scopes
     21332: object=/bin/sh [0]
     21332: scope 0: /bin/sh /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2
     21332:
     21332: object=linux-vdso.so.1 [0]
     21332: scope 0...

[Christian Ehrhardt  (paelzer)]

This was last built in hirsute, suspicious ...

Rebuilding as is in:
 https://launchpad.net/~paelzer/+archive/ubuntu/lp-1992566-golang-early-segfault
gave me a working binary just as my local rebuild had \o/

I can only assume that is some dark magic of the golang static build and an incompatibility dragging that old build from hirsute along so long?
Happy to be told more details as I'm curious about this ...

Since I don't understand the root cause yet I'm curious - could this be some kind of "breaks all go programs" issue that is hiding well so far?
I searched for all "<Hirsute" + "not rebuilt since focal" + "bdeps dh-golang" + "not just golang-* lib" programs to test them ...
Only "boohu" qualifies for all of that and it was not affected by the same AFAICS.

So I guess while I do not understand it yet the solution is simple.
We should rebuild this or flag this bug as fixed along any other coming upload?

[Bryce Harrington (bryce)]

> We should rebuild this or flag this bug as fixed along any other coming upload?

This received a rebuild for the new version uploaded to oracular. (Focal and Jammy might still need rebuilt.)