Snap applications segfault with new core20 (rev: 1015+)

[Łukasz Zemczak]

[Impact]

* Core20 snap built with updated glibc crashes snaps also bundling glibc.

[Test Plan]

* Build core20 snap with glibc in focal-proposed. Test a snap (which is not core20) bundling glibc:

TODO: install locally built core20?
snap install test-snapd-rsync-core20 --edge
snap run test-snapd-rsync-core20.rsync

[Where problems could occur]

* The previous glibc update (2.31-0ubuntu9.3) had a fix (LP: #1914044) that broke snaps bundling a previous version of glibc (2.31-0ubuntu9.2) due to them being incompatible. The fix of LP: #1914044 is reverted and 2.31-0ubuntu9.4 does not include changes incompatible with 2.31-0ubuntu9.2, thus the crash should not occur. No problems are expected.

[Original Bug Text]
It seems that with our new core20 in the beta channel all snaps seem to be segfaulting. We recently had a new glibc landed in focal-updates - might be related.

[Steve Langasek (vorlon)]

backtrace needed

[Ian Johnson (anonymouse67)]

With the test snap that uses core20 as it's base, test-snapd-rsync-core20 (installable on the edge channel), I see it segfaulting when running the snap on both a UC20 system with the core20 snap as a base snap, as well as on my groovy desktop:

https://pastebin.ubuntu.com/p/qbq86DYw5Q/

You can reproduce this with:
```
snap install core20 --beta || snap refresh core20 --beta
snap install test-snapd-rsync-core20 --edge
snap run --experimental-gdbserver test-snapd-rsync-core20.rsync

... you will see the gdb command to use to connect to the gdbserver running inside the snap's mount namespace
gdb -ex="target remote :33021" -ex=continue -ex="signal SIGCONT" # in another window
```

My gdb output seems to imply the segfault is coming from 0x000056287d57a2d4 in time@plt ? I haven't been able to load debug symbols for this gdb version into the snap mount namespace yet so I don't have more info, but presumably you could copy debug symbols into the snap's dir somewhere like $HOME/snap/test-snapd-rsync-core20/current/debug.sym and then load it from the gdb shell.

For reference, I've also attached the output of strace too in case that's more useful: https://pastebin.ubuntu.com/p/rpqKrnBHrg/

[Łukasz Zemczak (sil2100)]

Balint will be looking into it. For now we decided to pull the latest glibc update from focal-updates back to focal-proposed.

[Balint Reczey (rbalint)]

Thank you for the bug report.

The update has been reverted, please downgrade glibc binary packges to 2.31-0ubuntu9.2 until the new update becomes available.

The problem seems to be caused by the fix for LP: #1914044.

[Ioanna Alifieraki (joalif)]

@Balint not sure if you're already aware but the regression caused by LP: #1914044 may be causing
the problem in LP: #1867502 .
Earlier today people reported failed deployments with netinstall, autoinstalls etc,
which is now working again (I guess because 2.31-0ubuntu9.3 was pulled out of -updates).

[Balint Reczey (rbalint)]

@joalif Thanks, I've marked it as a duplicate of an other similar issue with the installer that i've already commented on.
BTW test-snapd-rsync-core20 does not work on Groovy and Hirsute either and I'm surprised that no one reported that yet.

[Balint Reczey (rbalint)]

Strangely I can reproduce the crash in a newly created multipass focal VM with the old glibc (2.31-0ubuntu9.2) from focal-updates.

[Balint Reczey (rbalint)]

OK, I've tested it in clean multipass VMs and test-snapd-rsync-core20.rsync does not work on Bionic, Focal and later. Interestingly it ships a local copy of libc6 inside that could be the problem and it worked on Focal for some time due to accidentally matching the host's libc6.

...
Reading target:/usr/lib/debug/snap/test-snapd-rsync-core20/11/lib/x86_64-linux-gnu//libc-2.31.so from remote target...

Program received signal SIGSEGV, Segmentation fault.
0x000055eb84b7b2d4 in time@plt ()
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007f070255d100 0x00007f070257f7c4 Yes (*) target:/lib64/ld-linux-x86-64.so.2
                                        No linux-vdso.so.1
0x00007f07025514f0 0x00007f07025557e8 Yes (*) target:/snap/test-snapd-rsync-core20/11/usr/lib/x86_64-linux-gnu/libacl.so.1
0x00007f0702543720 0x00007f070254a92d Yes (*) target:/snap/test-snapd-rsync-core20/11/usr/lib/x86_64-linux-gnu/libpopt.so.0
0x00007f0702374630 0x00007f07024e908f Yes (*) target:/snap/test-snapd-rsync-core20/11/lib/x86_64-linux-gnu/libc.so.6
(*): Shared library is missing debugging information.

For the record test-snapd-rsync-core18.rsync does not ship and internal libc copy and does work on all releases I tried (Bionic, Focal).

[Balint Reczey (rbalint)]

OK, so core20 (1015) bundles libc6 2.31-0ubuntu9.3 which has been removed from updates. Please build a new core20 with libc6 2.31-0ubuntu9.2 which is currently in focal-updates.

[Balint Reczey (rbalint)]

Core20 (1026) now works, but I believe shipping libc in test-snapd-rsync-core20, too, is not healthy and will break again when core20's glibc gets upgraded.

[Ian Johnson (anonymouse67)]

Sure, I was not aware test-snapd-rsync-core20 was shipping glibc, that is indeed not a good idea.

I went looking on my system for other snaps which experienced the crash, and it seems that every snap that ships glibc in it crashes with the beta channel of core20, but snaps that (properly) do not ship libc6 in them do not crash. For example these other well known snaps ship glibc in them:

* matterhorn
* okular
* htop

and some others that are perhaps less well known. So I think it is unfortunately a bit common to do this even though it is not advisable.

Sergio, do you know why these snaps would have libc6 staged in them? Matterhorn for example does not declare libc6 as a stage-package, yet it is listed as a primed-stage-packages in the manifest.yaml:

```snapcraft.yaml
    stage-packages:
      - libatomic1
      - libsecret-tools
      - libnotify-bin
      - xclip
```

```manifest.yaml
primed-stage-packages:
- libc6=2.31-0ubuntu9.2
```

[Sergio Schvezov (sergiusens)]

Hi Ian, thanks for raising this. Those would need a rebuild to be mostly ok, we had a release time bug which we have since fixed https://github.com/snapcore/snapcraft/commit/0bf7a2e6619b0037a50caeb49d28788c021d0921

If using Snapcraft 4.6.1 this should no longer be the issue for core20.

[Balint Reczey (rbalint)]

@anonymouse67 With glibc removed from the snaps other than core20 they should be working OK with core20 1016 shipping 2.31-0ubuntu9.3. Could you please confirm that?

[Ian Johnson (anonymouse67)]

Unfortunately I don't know how to easily remove glibc from the snaps in a way that would confirm that they work, I don't have time to manually build all of these snaps that are broken, I tried the basic thing of unpacking the snap and `rm -rf ./lib/x86_64-linux-gnu/libc-2.31.so ./lib/x86_64-linux-gnu/libc.so.6` and then repacking and installing these snaps, but then the still segfault and fail with:

$ snap run htop
*** stack smashing detected ***: terminated
Aborted (core dumped)

Which I don't know if that's because I didn't fully remove traces of glibc from the snap or if it's because beta version of core20 (snap revision 1015) is still broken.

I did try building the matterhorn snap since I found the source for it at https://github.com/popey/matterhorn-snap.git, but that doesn't seem to build at all.

Perhaps Sergio can help confirm if these snaps work if rebuilt without libc6 getting staged into the snap?

[Alkis Georgopoulos (alkisg)]

Hi, the libc update + removal caused the following issue:

1) On 2021-04-26, libc version 2.31-0ubuntu9.3 got uploaded to Ubuntu Focal.
2) Many people updated to it.
3) Two days later, on 2021-04-28, it got removed because it was causing the issues descripted in this bug report.
4) If any of the persons that updated, tries to install libc6-dev now, it says it's not installable as it depends on libc6=2.31-0ubuntu9.2, while 2.31-0ubuntu9.3 is installed.

In some cases they can downgrade with `apt install libc6=2.31-0ubuntu9.2`, but that's not always easy when additional dependencies are involved.

I believe a solution would be to re-upload 2.31-0ubuntu9.2 as 2.31-0ubuntu9.4. This would then not break snap, while allowing people to install libc6-dev, as it wouldn't have a lower version than in the archives anymore.

[Sergio Schvezov (sergiusens)]

The issue has been solved in Snapcraft, but any affected snap would need to rebuild

[Alkis Georgopoulos (alkisg)]

Another variant of the problem that I mentioned in comment #15. An ubuntu user just reported it on IRC:

$ apt install clang
The following packages have unmet dependencies:
 clang : Depends: clang-10 (>= 10~) but it is not going to be installed

That apt message is misleading, as the underlying error is the libc6 version:

$ dpkg -l | awk '$3=="2.31-0ubuntu9.3" { print $2, $3 }'
libc-bin 2.31-0ubuntu9.3
libc-dev-bin 2.31-0ubuntu9.3
libc6:amd64 2.31-0ubuntu9.3
libc6:i386 2.31-0ubuntu9.3
libc6-dbg:amd64 2.31-0ubuntu9.3
libc6-dev:amd64 2.31-0ubuntu9.3
locales 2.31-0ubuntu9.3

To be able to install clang, all these packages would have to be downgraded manually, with:

$ sudo apt install $(dpkg -l | awk '$3=="2.31-0ubuntu9.3" { print $2"=2.31-0ubuntu9.2" }')

The following packages will be DOWNGRADED:
  libc-bin libc-dev-bin libc6 libc6:i386 libc6-dbg libc6-dev locales

But we can't expect thousands of Ubuntu users to run that command manually.
Please do upload a new glibc 2.31-0ubuntu9.4 version, to amend this issue...

[Lmoal Laoml (lmoallaoml)]

Hello, apt was not able to resolve dependencies when trying to install clang on my end as well, and was saying that the package depends on libc6=2.31-0ubuntu9.2, which is not installed and can't be installed. I tried the fix @alkisg suggested and it works now. It downgraded 8-9 libc* packages, and I am now able to install clang

[Steve Langasek (vorlon)]

Hello Łukasz, or anyone else affected,

Accepted glibc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted glibc (2.31-0ubuntu9.4) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

snapd-glib/1.58-0ubuntu0.20.04.0 (armhf)
apt/2.0.6 (armhf)
libmath-mpfr-perl/4.13-1 (armhf)
art-nextgen-simulation-tools/20160605+dfsg-4 (armhf)
ruby-nokogiri/1.10.7+dfsg1-2build1 (armhf)
r-cran-rgdal/1.4-8-1build2 (armhf)
arrayfire/3.3.2+dfsg1-4ubuntu4 (armhf)
libpango-perl/1.227-3build1 (armhf)
libimage-sane-perl/5-1 (s390x)
ruby-bootsnap/1.4.6-1 (arm64)
mle/1.4.3-1 (ppc64el, arm64)
libsyntax-keyword-try-perl/0.11-1build1 (armhf)
awesome/4.3-4 (armhf)
cysignals/1.10.2+ds-4 (arm64)
gvfs/1.44.1-1ubuntu1 (ppc64el)
libuv1/1.34.2-1ubuntu1.3 (amd64)
bali-phy/3.4.1+dfsg-2build1 (arm64, s390x)
g10k/0.5.7-1 (armhf)
litl/0.1.9-7 (amd64)
ruby-libxml/3.1.0-2 (armhf)
ffmpeg/7:4.2.4-1ubuntu0.1 (armhf)
yorick/2.2.04+dfsg1-10 (ppc64el, s390x)
linux-ibm/5.4.0-1010.11 (amd64)
liborcus/0.15.3-3build2 (armhf)
node-nodedbi/1.0.13+dfsg-1build1 (amd64)
r-bioc-delayedarray/0.12.2+dfsg-1 (armhf)
postgresql-unit/7.2-2 (armhf)
python-freecontact/1.1-5build2 (armhf)
r-cran-rwave/2.4-8-2 (armhf)
libproc-fastspawn-perl/1.2-1build2 (armhf)
linux-hwe-5.11/5.11.0-44.48~20.04.2 (armhf)
foo2zjs/20171202dfsg0-4 (armhf)
r-cran-erm/1.0-0-1 (armhf)
libsys-cpuload-perl/0.03-8build5 (armhf)
libhttp-parser-xs-perl/0.17-1build5 (armhf)
php-luasandbox/3.0.3-2build2 (armhf)
pynfft/1.3.2-3build1 (armhf)
r-cran-processx/3.4.2-1 (ppc64el)
r-bioc-multtest/2.42.0-1 (armhf)
linux-hwe-5.13/5.13.0-23.23~20.04.2 (armhf)
python-blosc/1.7.0+ds1-2ubuntu2 (armhf)
gyoto/1.4.4-3 (armhf)
r-cran-sem/3.1.9-2build1 (armhf)
libtext-reflow-perl/1.17-1build3 (armhf)
python3.9/3.9.5-3ubuntu0~20.04.1 (armhf)
r-cran-samr/3.0-1 (armhf)
r-cran-dplyr/0.8.4-1 (armhf)
python3.8/3.8.10-0ubuntu1~20.04.2 (armhf)
findent/3.1.1-1build1 (armhf)
mercurial/5.3.1-1ubuntu1 (amd64)
libmemcached-libmemcached-perl/1.001801+dfsg-2build4 (armhf)
openbabel/3.0.0+dfsg-3ubuntu3 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Brian Murray (brian-murray)]

Hello Łukasz, or anyone else affected,

Accepted glibc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted glibc (2.31-0ubuntu9.5) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

garli/2.1-3build1 (armhf)
fpc/3.0.4+dfsg-23 (armhf)
lazarus/2.0.6+dfsg-3 (armhf)
libuv1/1.34.2-1ubuntu1.3 (i386)
ikiwiki-hosting/0.20180719-2 (armhf)
rtags/2.37-1 (amd64)
mercurial/5.3.1-1ubuntu1 (amd64, ppc64el)
foo2zjs/20171202dfsg0-4 (armhf)
frameworkintegration/5.68.0-0ubuntu1 (armhf)
hilive/2.0a-3build2 (arm64)
ruby-libxml/3.1.0-2 (s390x)
plasma-framework/5.68.0-0ubuntu1 (armhf)
feersum/1.407-2 (s390x)
r-bioc-delayedarray/0.12.2+dfsg-1 (armhf)
php-luasandbox/3.0.3-2build2 (ppc64el)
snapd-glib/1.58-0ubuntu0.20.04.0 (armhf)
bolt/0.8-4ubuntu1 (ppc64el)
threadweaver/5.68.0-0ubuntu1 (armhf)
python3.8/3.8.10-0ubuntu1~20.04.2 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!