stack smashing attack detected in bash host tab completion

[Seth Arnold]

Hello, this is a speculative bug report at best.

In some long-lived bash terminals, tab completion of hostnames on ping or ssh commands is printing the glibc stack smashing attempt error message:

$ ping goog*** stack smashing detected ***: terminated
^C
$ ssh local*** stack smashing detected ***: terminated
host ^C

I installed the glibc update 2.31-0ubuntu9.3 https://lists.ubuntu.com/archives/focal-changes/2021-April/024256.html earlier today. Shells started *after* this update work fine. Shells started before this update show this behaviour.

$ cat /proc/$$/maps
55f1986be000-55f1986eb000 r--p 00000000 00:1c 337406 /usr/bin/bash
55f1986eb000-55f19879c000 r-xp 0002d000 00:1c 337406 /usr/bin/bash
55f19879c000-55f1987d3000 r--p 000de000 00:1c 337406 /usr/bin/bash
55f1987d3000-55f1987d7000 r--p 00114000 00:1c 337406 /usr/bin/bash
55f1987d7000-55f1987e0000 rw-p 00118000 00:1c 337406 /usr/bin/bash
55f1987e0000-55f1987ea000 rw-p 00000000 00:00 0
55f19a673000-55f19b057000 rw-p 00000000 00:00 0 [heap]
7f29171e9000-7f29171ec000 r--p 00000000 00:1c 811498 /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7f29171ec000-7f29171f3000 r-xp 00003000 00:1c 811498 /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7f29171f3000-7f29171f5000 r--p 0000a000 00:1c 811498 /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7f29171f5000-7f29171f6000 r--p 0000b000 00:1c 811498 /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7f29171f6000-7f29171f7000 rw-p 0000c000 00:1c 811498 /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7f29171f7000-7f29171fd000 rw-p 00000000 00:00 0
7f2917210000-7f2917553000 r--p 00000000 00:1c 813840 /usr/lib/locale/locale-archive (deleted)
7f2917553000-7f2917556000 rw-p 00000000 00:00 0
7f2917556000-7f291757b000 r--p 00000000 00:1c 811482 /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7f291757b000-7f29176f3000 r-xp 00025000 00:1c 811482 /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7f29176f3000-7f291773d000 r--p 0019d000 00:1c 811482 /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7f291773d000-7f291773e000 ---p 001e7000 00:1c 811482 /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7f291773e000-7f2917741000 r--p 001e7000 00:1c 811482 /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7f2917741000-7f2917744000 rw-p 001ea000 00:1c 811482 /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7f2917744000-7f2917748000 rw-p 00000000 00:00 0
7f2917748000-7f2917749000 r--p 00000000 00:1c 811484 /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7f2917749000-7f291774b000 r-xp 00001000 00:1c 811484 /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7f291774b000-7f291774c000 r--p 00003000 00:1c 811484 /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7f291774c000-7f291774d000 r--p 00003000 00:1c 811484 /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7f291774d000-7f291774e000 rw-p 00004000 00:1c 811484 /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7f291774e000-7f291775c000 r--p 00000000 00:1c 659440 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f291775c000-7f291776b000 r-xp 0000e000 00:1c 659440 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f291776b000-7f2917779000 r--p 0001d000 00:1c 659440 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f2917779000-7f291777d000 r--p 0002a000 00:1c 659440 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f291777d000-7f291777e000 rw-p 0002e000 00:1c 659440 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f291777e000-7f2917780000 rw-p 00000000 00:00 0
7f291778c000-7f2917793000 r--s 00000000 00:1c 813296 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache (deleted)
7f2917793000-7f2917794000 r--p 00000000 00:1c 811474 /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7f2917794000-7f29177b7000 r-xp 00001000 00:1c 811474 /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7f29177b7000-7f29177bf000 r--p 00024000 00:1c 811474 /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7f29177c0000-7f29177c1000 r--p 0002c000 00:1c 811474 /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7f29177c1000-7f29177c2000 rw-p 0002d000 00:1c 811474 /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7f29177c2000-7f29177c3000 rw-p 00000000 00:00 0
7ffd864bb000-7ffd864dc000 rw-p 00000000 00:00 0 [stack]
7ffd865b4000-7ffd865b7000 r--p 00000000 00:00 0 [vvar]
7ffd865b7000-7ffd865b8000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
$

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: libc6 2.31-0ubuntu9.3
ProcVersionSignature: Ubuntu 5.4.0-71.79-generic 5.4.101
Uname: Linux 5.4.0-71-generic x86_64
NonfreeKernelModules: lkp_Ubuntu_5_4_0_71_79_generic_76 zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
CasperMD5CheckResult: skip
Date: Tue Apr 27 23:30:08 2021
ProcEnviron:
 TERM=rxvt-unicode-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: glibc
UpgradeStatus: Upgraded to focal on 2020-01-24 (459 days ago)

[Seth Arnold (seth-arnold)]

Possibly a duplicate of bug LP: #1926355 https://bugs.launchpad.net/bugs/1926355

[Seth Arnold (seth-arnold)]

I couldn't install the debug symbols:

 bash-dbgsym : Depends: bash (= 5.0-6ubuntu1) but 5.0-6ubuntu1.1 is to be installed

And I had to add a Package: bash line to my crash file..

Here's the frames that look most likely related:

#13 0x00007f29177a85fa in _dl_find_dso_for_object () from /lib64/ld-linux-x86-64.so.2
No symbol table info available.
#14 0x000055f19a675880 in ?? ()
No symbol table info available.
#15 0x00007ffd864d6140 in ?? ()
No symbol table info available.
#16 0x00007f2980000002 in ?? ()
No symbol table info available.
#17 0x00007f291769c62c in nss_load_library (ni=0x0) at nsswitch.c:359
        shlen = <error reading variable shlen (Cannot access memory at address 0xffffffb7)>
        saved_errno = 1
        shlib_name = <error reading variable shlib_name (Cannot access memory at address 0xffffffb7)>
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[Balint Reczey (rbalint)]

Thank you for the bug report.

The update has been reverted, please downgrade glibc binary packges to 2.31-0ubuntu9.2 until the new update becomes available.

The problem seems to be caused by the fix for LP: #1914044.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Haw Loeung (hloeung)]

We're also seeing this with rsyncd:

| Apr 28 09:35:04 behaim rsync[2152929]: *** stack smashing detected ***: terminated

From apt history logs:

| Upgrade: libldap-2.4-2:amd64 (2.4.49+dfsg-2ubuntu1.7, 2.4.49+dfsg-2ubuntu1.8), libc6-dev:amd64 (2.31-0ubuntu9.2, 2.31-0ubuntu9.3), grub-common:amd64 (2.04-1ubuntu26.9, 2.04-1ubuntu26.11), python3-pip:amd64 (20.0.2-5ubuntu1.1, 20.0.2-5ubuntu1.3), libc6:amd64 (2.31-0ubuntu9.2,
2.31-0ubuntu9.3)...

[Balint Reczey (rbalint)]

There is a WIP branch to prevent upgrades from 2.31-0ubuntu9.3 and cause crashes again on that path: https://code.launchpad.net/~rbalint/ubuntu/+source/glibc/+git/glibc/+ref/ubuntu/focal

[Michael Hudson-Doyle (mwhudson)]

So I don't know exactly what was going on with sarnold's machine but I now finally understand why the 0ubuntu9.3 update caused problems:

The tls accounting patch added a glibc tunable (https://www.gnu.org/software/libc/manual/html_node/Tunables.html). A tunable is defined internally as a name and a type (and some other data) but during the build it also gets assigned an ID and unfortunately the tunable added by the tls accounting patch ends changing the ID of the glibc.pthread.mutex_spin_count tunable. The problems occur when you have a new dynamic linker / ld.so but an old libpthread.so: libpthread.so's _init function calls get_tunable with the ID for glibc.pthread.mutex_spin_count, but get_tunable is implemented in ld.so, where this ID corresponds to the new glibc.rtld.nns tunable. The type for glibc.pthread.mutex_spin_count is int32 and the type for glibc.rtld.nns is size_t, so when get_tunable writes the value into the pointer it is passed, it does indeed smash the stack. Even if this doesn't happen, libpthread might well misbehave in all sorts of ways if gets back values appropriate for glibc.rtld.nns when it's expecting values for glibc.pthread.mutex_spin_count.

So this explains the behaviour seen in bug 1926355, completely. What I don't understand wrt this bug is that "new ld.so / old libpthread.so" should be a very temporary situation during an upgrade. I guess a process that has the old ld.so loaded might dlopen the new libpthread.so and experience a similar issue, although dlopening libpthread isn't really a think that works aiui. But it could be a similar problem with some other library.

Unfortunately, this means that upgrades from 0ubuntu9.3 to 0ubuntu9.4 are vulnerable to the same issue.

[Steve Langasek (vorlon)]

Hello Seth, or anyone else affected,

Accepted glibc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Seth Arnold (seth-arnold)]

What kinds of tests are we hoping for, here? From people who never backed out the broken glibc manually? (This might be a difficult group to reach.)

Or from just anyone, to try to find out if the revert and new fixes were done correctly?

Thanks

[Michael Hudson-Doyle (mwhudson)]

I'm more interested in people updating from the version currently in updates to the version currently in proposed.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted glibc (2.31-0ubuntu9.4) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

snapd-glib/1.58-0ubuntu0.20.04.0 (armhf)
apt/2.0.6 (armhf)
libmath-mpfr-perl/4.13-1 (armhf)
art-nextgen-simulation-tools/20160605+dfsg-4 (armhf)
ruby-nokogiri/1.10.7+dfsg1-2build1 (armhf)
r-cran-rgdal/1.4-8-1build2 (armhf)
arrayfire/3.3.2+dfsg1-4ubuntu4 (armhf)
libpango-perl/1.227-3build1 (armhf)
libimage-sane-perl/5-1 (s390x)
ruby-bootsnap/1.4.6-1 (arm64)
mle/1.4.3-1 (ppc64el, arm64)
libsyntax-keyword-try-perl/0.11-1build1 (armhf)
awesome/4.3-4 (armhf)
cysignals/1.10.2+ds-4 (arm64)
gvfs/1.44.1-1ubuntu1 (ppc64el)
libuv1/1.34.2-1ubuntu1.3 (amd64)
bali-phy/3.4.1+dfsg-2build1 (arm64, s390x)
g10k/0.5.7-1 (armhf)
litl/0.1.9-7 (amd64)
ruby-libxml/3.1.0-2 (armhf)
ffmpeg/7:4.2.4-1ubuntu0.1 (armhf)
yorick/2.2.04+dfsg1-10 (ppc64el, s390x)
linux-ibm/5.4.0-1010.11 (amd64)
liborcus/0.15.3-3build2 (armhf)
node-nodedbi/1.0.13+dfsg-1build1 (amd64)
r-bioc-delayedarray/0.12.2+dfsg-1 (armhf)
postgresql-unit/7.2-2 (armhf)
python-freecontact/1.1-5build2 (armhf)
r-cran-rwave/2.4-8-2 (armhf)
libproc-fastspawn-perl/1.2-1build2 (armhf)
linux-hwe-5.11/5.11.0-44.48~20.04.2 (armhf)
foo2zjs/20171202dfsg0-4 (armhf)
r-cran-erm/1.0-0-1 (armhf)
libsys-cpuload-perl/0.03-8build5 (armhf)
libhttp-parser-xs-perl/0.17-1build5 (armhf)
php-luasandbox/3.0.3-2build2 (armhf)
pynfft/1.3.2-3build1 (armhf)
r-cran-processx/3.4.2-1 (ppc64el)
r-bioc-multtest/2.42.0-1 (armhf)
linux-hwe-5.13/5.13.0-23.23~20.04.2 (armhf)
python-blosc/1.7.0+ds1-2ubuntu2 (armhf)
gyoto/1.4.4-3 (armhf)
r-cran-sem/3.1.9-2build1 (armhf)
libtext-reflow-perl/1.17-1build3 (armhf)
python3.9/3.9.5-3ubuntu0~20.04.1 (armhf)
r-cran-samr/3.0-1 (armhf)
r-cran-dplyr/0.8.4-1 (armhf)
python3.8/3.8.10-0ubuntu1~20.04.2 (armhf)
findent/3.1.1-1build1 (armhf)
mercurial/5.3.1-1ubuntu1 (amd64)
libmemcached-libmemcached-perl/1.001801+dfsg-2build4 (armhf)
openbabel/3.0.0+dfsg-3ubuntu3 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Michael Hudson-Doyle (mwhudson)]

I tested an upgrade from 0ubuntu9.2 to 0ubuntu9.4 in a vm and could not reproduce this problem.

[Brian Murray (brian-murray)]

Hello Seth, or anyone else affected,

Accepted glibc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted glibc (2.31-0ubuntu9.5) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

garli/2.1-3build1 (armhf)
fpc/3.0.4+dfsg-23 (armhf)
lazarus/2.0.6+dfsg-3 (armhf)
libuv1/1.34.2-1ubuntu1.3 (i386)
ikiwiki-hosting/0.20180719-2 (armhf)
rtags/2.37-1 (amd64)
mercurial/5.3.1-1ubuntu1 (amd64, ppc64el)
foo2zjs/20171202dfsg0-4 (armhf)
frameworkintegration/5.68.0-0ubuntu1 (armhf)
hilive/2.0a-3build2 (arm64)
ruby-libxml/3.1.0-2 (s390x)
plasma-framework/5.68.0-0ubuntu1 (armhf)
feersum/1.407-2 (s390x)
r-bioc-delayedarray/0.12.2+dfsg-1 (armhf)
php-luasandbox/3.0.3-2build2 (ppc64el)
snapd-glib/1.58-0ubuntu0.20.04.0 (armhf)
bolt/0.8-4ubuntu1 (ppc64el)
threadweaver/5.68.0-0ubuntu1 (armhf)
python3.8/3.8.10-0ubuntu1~20.04.2 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!