Snap applications segfault with new core20 (rev: 1015+)

Bug #1926355 reported by Łukasz Zemczak on 2021-04-27
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glibc (Ubuntu)
Balint Reczey

Bug Description

It seems that with our new core20 in the beta channel all snaps seem to be segfaulting. We recently had a new glibc landed in focal-updates - might be related.

Steve Langasek (vorlon) wrote :

backtrace needed

Changed in glibc (Ubuntu):
status: New → Incomplete
Ian Johnson (anonymouse67) wrote :

With the test snap that uses core20 as it's base, test-snapd-rsync-core20 (installable on the edge channel), I see it segfaulting when running the snap on both a UC20 system with the core20 snap as a base snap, as well as on my groovy desktop:

You can reproduce this with:
snap install core20 --beta || snap refresh core20 --beta
snap install test-snapd-rsync-core20 --edge
snap run --experimental-gdbserver test-snapd-rsync-core20.rsync

... you will see the gdb command to use to connect to the gdbserver running inside the snap's mount namespace
gdb -ex="target remote :33021" -ex=continue -ex="signal SIGCONT" # in another window

My gdb output seems to imply the segfault is coming from 0x000056287d57a2d4 in time@plt ? I haven't been able to load debug symbols for this gdb version into the snap mount namespace yet so I don't have more info, but presumably you could copy debug symbols into the snap's dir somewhere like $HOME/snap/test-snapd-rsync-core20/current/debug.sym and then load it from the gdb shell.

For reference, I've also attached the output of strace too in case that's more useful:

Changed in glibc (Ubuntu):
status: Incomplete → New
Łukasz Zemczak (sil2100) wrote :

Balint will be looking into it. For now we decided to pull the latest glibc update from focal-updates back to focal-proposed.

Balint Reczey (rbalint) wrote :

Thank you for the bug report.

The update has been reverted, please downgrade glibc binary packges to 2.31-0ubuntu9.2 until the new update becomes available.

The problem seems to be caused by the fix for LP: #1914044.

Balint Reczey (rbalint) on 2021-04-28
Changed in glibc (Ubuntu):
assignee: nobody → Balint Reczey (rbalint)
Ioanna Alifieraki (joalif) wrote :

@Balint not sure if you're already aware but the regression caused by LP: #1914044 may be causing
the problem in LP: #1867502 .
Earlier today people reported failed deployments with netinstall, autoinstalls etc,
which is now working again (I guess because 2.31-0ubuntu9.3 was pulled out of -updates).

Balint Reczey (rbalint) wrote :

@joalif Thanks, I've marked it as a duplicate of an other similar issue with the installer that i've already commented on.
BTW test-snapd-rsync-core20 does not work on Groovy and Hirsute either and I'm surprised that no one reported that yet.

tags: added: regression-update
Balint Reczey (rbalint) wrote :

Strangely I can reproduce the crash in a newly created multipass focal VM with the old glibc (2.31-0ubuntu9.2) from focal-updates.

Balint Reczey (rbalint) wrote :

OK, I've tested it in clean multipass VMs and test-snapd-rsync-core20.rsync does not work on Bionic, Focal and later. Interestingly it ships a local copy of libc6 inside that could be the problem and it worked on Focal for some time due to accidentally matching the host's libc6.

Reading target:/usr/lib/debug/snap/test-snapd-rsync-core20/11/lib/x86_64-linux-gnu// from remote target...

Program received signal SIGSEGV, Segmentation fault.
0x000055eb84b7b2d4 in time@plt ()
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007f070255d100 0x00007f070257f7c4 Yes (*) target:/lib64/
0x00007f07025514f0 0x00007f07025557e8 Yes (*) target:/snap/test-snapd-rsync-core20/11/usr/lib/x86_64-linux-gnu/
0x00007f0702543720 0x00007f070254a92d Yes (*) target:/snap/test-snapd-rsync-core20/11/usr/lib/x86_64-linux-gnu/
0x00007f0702374630 0x00007f07024e908f Yes (*) target:/snap/test-snapd-rsync-core20/11/lib/x86_64-linux-gnu/
(*): Shared library is missing debugging information.

For the record test-snapd-rsync-core18.rsync does not ship and internal libc copy and does work on all releases I tried (Bionic, Focal).

Changed in glibc (Ubuntu):
status: New → Invalid
Changed in glibc (Ubuntu Groovy):
status: New → Invalid
Changed in glibc (Ubuntu Hirsute):
status: New → Invalid
Balint Reczey (rbalint) on 2021-05-05
Changed in glibc (Ubuntu):
status: Invalid → New
Changed in glibc (Ubuntu Groovy):
status: Invalid → New
Changed in glibc (Ubuntu Hirsute):
status: Invalid → New
Balint Reczey (rbalint) wrote :

OK, so core20 (1015) bundles libc6 2.31-0ubuntu9.3 which has been removed from updates. Please build a new core20 with libc6 2.31-0ubuntu9.2 which is currently in focal-updates.

Balint Reczey (rbalint) wrote :

Core20 (1026) now works, but I believe shipping libc in test-snapd-rsync-core20, too, is not healthy and will break again when core20's glibc gets upgraded.

Ian Johnson (anonymouse67) wrote :

Sure, I was not aware test-snapd-rsync-core20 was shipping glibc, that is indeed not a good idea.

I went looking on my system for other snaps which experienced the crash, and it seems that every snap that ships glibc in it crashes with the beta channel of core20, but snaps that (properly) do not ship libc6 in them do not crash. For example these other well known snaps ship glibc in them:

* matterhorn
* okular
* htop

and some others that are perhaps less well known. So I think it is unfortunately a bit common to do this even though it is not advisable.

Sergio, do you know why these snaps would have libc6 staged in them? Matterhorn for example does not declare libc6 as a stage-package, yet it is listed as a primed-stage-packages in the manifest.yaml:

      - libatomic1
      - libsecret-tools
      - libnotify-bin
      - xclip

- libc6=2.31-0ubuntu9.2

Sergio Schvezov (sergiusens) wrote :

Hi Ian, thanks for raising this. Those would need a rebuild to be mostly ok, we had a release time bug which we have since fixed

If using Snapcraft 4.6.1 this should no longer be the issue for core20.

Balint Reczey (rbalint) wrote :

@anonymouse67 With glibc removed from the snaps other than core20 they should be working OK with core20 1016 shipping 2.31-0ubuntu9.3. Could you please confirm that?

Ian Johnson (anonymouse67) wrote :

Unfortunately I don't know how to easily remove glibc from the snaps in a way that would confirm that they work, I don't have time to manually build all of these snaps that are broken, I tried the basic thing of unpacking the snap and `rm -rf ./lib/x86_64-linux-gnu/ ./lib/x86_64-linux-gnu/` and then repacking and installing these snaps, but then the still segfault and fail with:

$ snap run htop
*** stack smashing detected ***: terminated
Aborted (core dumped)

Which I don't know if that's because I didn't fully remove traces of glibc from the snap or if it's because beta version of core20 (snap revision 1015) is still broken.

I did try building the matterhorn snap since I found the source for it at, but that doesn't seem to build at all.

Perhaps Sergio can help confirm if these snaps work if rebuilt without libc6 getting staged into the snap?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers