Mantic s390x install fails: grep-status: *** stack smashing detected ***: terminated

[Paride Legovini]

mantic-live-server-s390x ISOs (serial: 20230918) are failing to install, as subiquity crashes with the error reported at the end of this message. By jumping to a shell in the installer live system, I found of that the culprit is grep-status failing to run:

root@ubuntu-server:/# grep-status
*** stack smashing detected ***: terminated
Aborted (core dumped)

What we expect is:

$ grep-status
grep-status: unexpected end of filter.

From the test history I can tell that ISOs with serials:

 * Up to 20230915 are working
 * From 20230916 up to (at least) 20230918 are failing.

--------------------

2023-09-18 11:59:52,651 DEBUG subiquitycore.utils:132 arun_command ['grep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status'] exited with code -6
2023-09-18 11:59:52,651 ERROR root:30 finish: subiquity/Install/install/curtin_install: FAIL: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.
2023-09-18 11:59:52,651 DEBUG subiquity.common.errorreport:394 generating crash report
2023-09-18 11:59:52,695 INFO subiquity.common.errorreport:415 saving crash report 'install failed crashed with CalledProcessError' to /var/crash/1695038392.651862621.install_fail.crash
2023-09-18 11:59:52,695 ERROR root:30 finish: subiquity/Install/install: FAIL: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.
2023-09-18 11:59:52,695 INFO root:30 start: subiquity/ErrorReporter/1695038392.651862621.install_fail/add_info:
2023-09-18 11:59:52,699 ERROR subiquity.server.server:415 top level error
Traceback (most recent call last):
  File "/snap/subiquity/5108/lib/python3.10/site-packages/subiquity/server/controllers/shutdown.py", line 74, in _wait_install
    await self.app.controllers.Install.install_task
subprocess.CalledProcessError: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.
2023-09-18 11:59:52,701 ERROR subiquity.server.server:415 top level error
Traceback (most recent call last):
  File "/snap/subiquity/5108/lib/python3.10/site-packages/subiquity/server/controllers/shutdown.py", line 74, in _wait_install
    await self.app.controllers.Install.install_task
subprocess.CalledProcessError: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.

[Paride Legovini (paride)]

The described behavior happens when installing a QEMU VM, the host system is Jammy running on an LPAR.

[Paride Legovini (paride)]

@fheimes it would be interesting to know if the same happens on a "bare" lpar (no qemu).

[Olivier Gayot (ogayot)]

Some binaries under /snap/subiquity/.../usr/bin on s390x use the libc from within the snap (they work properly) whereas others use the libc from outside the snap (/lib/s390x-linux-gnu) and they fail.

root@ubuntu-server:/# ldd /snap/subiquity/current/usr/bin/zstd
 linux-vdso64.so.1 (0x000003ffe1ead000)
 libz.so.1 => /snap/core22/current/lib/s390x-linux-gnu/libz.so.1 (0x000003ffb6580000)
 liblzma.so.5 => /snap/core22/current/lib/s390x-linux-gnu/liblzma.so.5 (0x000003ffb6500000)
 liblz4.so.1 => /snap/core22/current/lib/s390x-linux-gnu/liblz4.so.1 (0x000003ffb6480000)
 libc.so.6 => /snap/core22/current/lib/s390x-linux-gnu/libc.so.6 (0x000003ffb6280000)
 /snap/core22/current/lib/ld64.so.1 => /lib/ld64.so.1 (0x000003ffb6700000)
root@ubuntu-server:/# /snap/subiquity/current/usr/bin/zstd
stdin is a console, aborting
root@ubuntu-server:/# ldd /snap/subiquity/current/usr/bin/distro-info
 linux-vdso64.so.1 (0x000003ffdd2c8000)
 libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ffa3b80000)
 /snap/core22/current/lib/ld64.so.1 => /lib/ld64.so.1 (0x000003ffa3f00000)
root@ubuntu-server:/# /snap/subiquity/current/usr/bin/distro-info
*** stack smashing detected ***: terminated
Aborted (core dumped)

This looks somewhat related to snapcraft patchelf. See binary patching at https://snapcraft.io/docs/linters-classic

[Olivier Gayot (ogayot)]

Excerpt from build log from Sept 11 [1]:

Patch ELF file: 'usr/bin/grep-dctrl'
  Interpreter='/snap/core22/current/lib/ld64.so.1'
  Current rpath=[]
  Proposed rpath=['/snap/core22/current/lib/s390x-linux-gnu']
Patch ELF file: 'usr/bin/join-dctrl'

Excerpt from build log from Sept 15 [2]:

Patch ELF file: 'usr/bin/grep-dctrl'
  Interpreter='/snap/core22/current/lib/ld64.so.1'
Patch ELF file: 'usr/bin/join-dctrl'

[1] https://launchpadlibrarian.net/686744619/buildlog_snap_ubuntu_jammy_s390x_subiquity_BUILDING.txt.gz
[2] https://launchpadlibrarian.net/687486601/buildlog_snap_ubuntu_jammy_s390x_subiquity_BUILDING.txt.gz

[Sergio Schvezov (sergiusens)]

May I see the snapcraft.yaml for this? Is grep-status properly patched?

[Michael Hudson-Doyle (mwhudson)]

It's here https://github.com/canonical/subiquity/blob/main/snapcraft.yaml

and it's clear the files have been patchelfed as they have the interpreter set to /snap/core22/... but for some reason the ones from the latest build don't have rpath set:

ubuntu@juju-fa3835-porterboxen-37:~$ readelf -l ./subiquity-5095/usr/bin/grep-status | grep interpreter
      [Requesting program interpreter: /snap/core22/current/lib/ld64.so.1]
ubuntu@juju-fa3835-porterboxen-37:~$ readelf -l ./subiquity-5108/usr/bin/grep-status | grep interpreter
      [Requesting program interpreter: /snap/core22/current/lib/ld64.so.1]
ubuntu@juju-fa3835-porterboxen-37:~$ readelf -d ./subiquity-5095/usr/bin/grep-status | grep RPATH
 0x000000000000000f (RPATH) Library rpath: [/snap/core22/current/lib/s390x-linux-gnu]
ubuntu@juju-fa3835-porterboxen-37:~$ readelf -d ./subiquity-5108/usr/bin/grep-status | grep RPATH
ubuntu@juju-fa3835-porterboxen-37:~$

[Michael Hudson-Doyle (mwhudson)]

I assume the cause for this is going to turn out to be that there was a glibc SRU released between the two snap builds:

Get:4 http://ftpmaster.internal/ubuntu jammy-updates/main s390x libc6-dev s390x 2.35-0ubuntu3.1 [1499 kB]
Get:4 http://ftpmaster.internal/ubuntu jammy-updates/main s390x libc6-dev s390x 2.35-0ubuntu3.3 [1501 kB]

I noticed that snapcraft tries a bunch of different ways to determine the dependencies of an elf object https://github.com/snapcore/snapcraft/blob/main/snapcraft/elf/_elf_file.py#L416-L439 and this bug in particular being fixed https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1995362 might influence which one succeeds.

[Michael Hudson-Doyle (mwhudson)]

Or possibly its https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/2007599 making something fail that used to succeed as that patch probably breaks using the new libc6.so with the old ld.so or something along those lines.

[Frank Heimes (fheimes)]

Hmm, but this (LP#2007599) is also included in Jammy and Lunar, and as of today I am only aware that this issue happens with a mantic install.

[Olivier Gayot (ogayot)]

@fheimes The bug would likely occur as well on recent s390x builds for other series. Currently though, only the devel series (i.e., mantic) uses grep-status. I wouldn't be surprised if you ran into the same issue after doing a snap refresh latest/edge on a 22.04 ISO for instance.

[Frank Heimes (fheimes)]

Ok, that could be of course - if the issue came in _after_ 22.04.3 got released, since the .3 was significantly tested and didn't showed that issue - but I see.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2036424

[Michael Hudson-Doyle (mwhudson)]

An acceptable workaround seems to be to build the snap with edge core22, which is what we'll do for beta.

[Dan Bungert (dbungert)]

Rebuilt snap on beta and stable/ubuntu-23.10 channels. It does appear to have patchelf'ed correctly, per the edge core22 suggestion.

[Michael Hudson-Doyle (mwhudson)]

> Hmm, but this (LP#2007599) is also included in Jammy and Lunar, and as of today I am only aware that this issue happens with a mantic install.

Yes but the issue with a missing path is that libraries are using ld.so from the core snap and libc6 from the host system. If the core snap and host system are using the same version of glibc this isn't going to cause any problems!

[Frank Heimes (fheimes)]

Just wanted to let you know that I did a few installation (z/VM and LPAR) based on the respin from the 20th (20092023) and it looks good - didn't bumped into this anymore.

[Dan Bungert (dbungert)]

@Frank - I think this problem is resolved with stable core22. The build logs look as expected. Would you mind testing the build found at subiquity channel beta/s390x-stable-core22?

[Frank Heimes (fheimes)]

@dbungert, sorry for the delay,
I just tried beta/s390x-stable-core22 (based on the image 2023-10-05, but of course updated the subiquity):
  subiquity 22.02.2+git1772.4a76b65e 5196
  snap refresh subiquity --channel=beta/s390x-stable-core22 --ignore-running
  subiquity 22.02.2+git1793.3841511f 5227 latest/beta/…
but I unfortunately still face the grep-status issue on an LPAR install:

 2023-10-08 20:11:02,365 DEBUG subiquitycore.utils:121 arun_command called: ['gr
ep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'in
stalled', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/stat
us']
 2023-10-08 20:11:02,481 DEBUG subiquitycore.utils:135 arun_command ['grep-statu
s', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed'
, '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status'] exi
ted with code -6
 2023-10-08 20:11:02,481 ERROR root:30 finish: subiquity/Install/install/curtin_
install: FAIL: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-imag
e', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names'
, '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.
 2023-10-08 20:11:02,481 DEBUG subiquity.common.errorreport:394 generating crash report
 2023-10-08 20:11:02,484 INFO subiquity.common.errorreport:415 saving crash report 'install failed crashed with CalledProcessError' to /var/crash/1696795862.481621742.install_fail.crash
 2023-10-08 20:11:02,484 ERROR root:30 finish: subiquity/Install/install: FAIL: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.
 2023-10-08 20:11:02,484 INFO root:30 start: subiquity/ErrorReporter/1696795862.481621742.install_fail/add_info:
...

These are all the snap versions that I see during the installation:
Name Version Rev Tracking Publisher Notes
core22 20230801 865 latest/stable canonical** base
lxd 5.18-db8c6f9 25850 latest/stable/… canonical** -
snapd 2.60.3 20099 latest/stable canonical** snapd
subiquity 22.02.2+git1793.3841511f 5227 latest/beta/… canonical** classic

I've also attached the logs and crash files.

[Dan Bungert (dbungert)]

Thanks Frank.

I'm marking the "Affects Mantic" as fix released, as we have a plan. The builds today continue to use edge channel core22, like before. Keeping the issue open in general to track a fix to move back to stable channel core22.