Segfault in ld-2.19.so while starting Steam after upgrade to 3.13.0-59.98

[Stefan Bader]

The previous kernel 3.13.0-58.97 does not show the issue when booting back into it. Currently only noticed on starting Steam but there its is reproducable every time on X230 laptop (Intel graphics). Might be related to Steam using 32bit libraries.
Attaching some log and dump that steam produces but fails to upload (wherever).

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-59-generic 3.13.0-59.98
ProcVersionSignature: Ubuntu 3.13.0-59.98-generic 3.13.11-ckt22
Uname: Linux 3.13.0-59-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.11
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: smb 2678 F.... pulseaudio
CurrentDesktop: Unity
Date: Tue Jul 28 21:42:17 2015
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=7aacd0c3-12d4-4f50-baf7-e1fd8f871f95
InstallationDate: Installed on 2013-07-25 (733 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64 (20130424)
MachineType: LENOVO 2324CTO
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-59-generic.efi.signed root=UUID=5d3281d3-0142-4039-bb9b-28653778aca0 ro quiet splash mmc-core.removable=0 nomdmonddf nomdmonisw vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-59-generic N/A
 linux-backports-modules-3.13.0-59-generic N/A
 linux-firmware 1.127.14
SourcePackage: linux
UpgradeStatus: Upgraded to trusty on 2014-04-29 (455 days ago)
dmi.bios.date: 04/30/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: G2ET94WW (2.54 )
dmi.board.asset.tag: Not Available
dmi.board.name: 2324CTO
dmi.board.vendor: LENOVO
dmi.board.version: Win8 Pro DPK TPG
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrG2ET94WW(2.54):bd04/30/2013:svnLENOVO:pn2324CTO:pvrThinkPadX230:rvnLENOVO:rn2324CTO:rvrWin8ProDPKTPG:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2324CTO
dmi.product.version: ThinkPad X230
dmi.sys.vendor: LENOVO

[Stefan Bader (smb)]

From the dump this part of steam looks to be involved but not sure how this relates to the segfaults. Each attempt creates 3 (I think of those).

Assert( Assertion Failed: CApplicationManager::GetMountVolume: invalid index ):/home/buildbot/buildslave_steam/steam_rel_client_ubuntu12_linux/build/src/clientdll/applicationmanager.cpp:3117

[Brad Figg (brad-figg)]

This change was made by a bot.

[Ool (0ol)]

I post this work around first in a duplicate bug report, so I put it here to:

I change the kernel version of my Ubuntu Trusty 14.04.02 from 3.13.0-59 to 3.16.0-45 with the LTS enable stack:
https://wiki.ubuntu.com/Kernel/LTSEnablementStack
no more segfault :)

[Brad Figg (brad-figg)]

This change was made by a bot.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stefan Bader (smb)]

With the latest 3.16-lts-trusty there is no problem.

[Felipe Castillo (fcastillo.ec)]

@Stefan I'm not quite sure what your statement means? I'm using Ubuntu Trusty and I'm having this problem. Is there a different kernel for trusty than pre-installed one?
Is this also a workaround for the issue, or would it be better to just use kernel -58?

[Daniel Convissor (convissor)]

I'm seeing this too trying to run netflix-desktop under the 3.13.0-59 kernel. (Which worked fine until today.)

@fcastillo.ec: Stefan was telling us that the LTS Enablement Stack (suggested by Ool) worked for him. Sure, you can use the 58 kernel if that works for you, but do be aware that the 59 kernel contains security fixes.

The LTS Enablement Stack worked for me, but do be aware, it removed several other packages I had installed (netflix-desktop among them). Upon reinstalling netflix-desktop, I'm back in business.

[Stefan Bader (smb)]

The Vivid LTS kernel is ok as well.

[Stefan Bader (smb)]

As for work-arounds: the simplest and quickest for the moment would be to boot the previous kernel, but as it was said that opens up some security issue that the current upload tried to fix. We work on fixing the regression as soon as possible.

Moving to a HWE stack seems to be another option but one which comes with a lot more change to the system. And the more change the more risk something may break. Both HWE kernels look to work correctly with the security fix applied.

[Olivier Debon (olivier-debon)]

Another weird workaround I discovered while investigating, just steam this way:
~user # strace -f -o/dev/null steam

It slow down UI, so it could be a race condition on steam client, though it ran fine before upgrading ubuntu.

If that helps.

[Adam Conrad (adconrad)]

A fix for this is building right now and, if all goes well, should be released in 8 to 12 hours.

[ThePhilips (thephilips)]

Just a ping. After update to the *-59 kernel, several internal applications started failing in system() library call: the shell (/bin/sh -c) was crashing shortly after exec() with a segmentation fault. Similarly to comment #15, the applications works if started using the 'strace -f'.

[georg (georg-g)]

I can confirm segfaults after updating the kernel to 3.13.0-59 on various 64 Bit machines. Its related to shelling out commands from 32 bit executables. In my case its a proprietary program, but its reproducible with the following steps:

    $ apt-get install gcc-multilib
    $ cat test.c
    #include "stdlib.h"
    int main()
    {
        system("/bin/echo Hello World");
    }
    $ gcc -m64 test.c
    Hello World
    $ gcc -m32 test.c
    $ ./a.out # Program is crashing here
    $ dmesg|tail -n 1
    [ 102.260840] sh[2283]: segfault at 3dbb92d0 ip 000000003dbb92d0 sp 00000000b366e850 error 14 in ld-2.19.so[7f573dbb8000+23000]

[Jerre Cope (jerre)]

Also affected with this bug after kernel upgrade with

error 14 in ld-2.19.so

application affected is a Business Basic interpreter from throughbredsoftware.com I don't have the source to this but the problem seems to relate to exclusive file locks

Reverting to the prior kernel resolves the problem.

[Norman Wilson (norma7)]

Here is a simpler example program:

#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/types.h>
#include <wait.h>

int
main(argc, argv)
int argc;
char **argv;
{
 int pid, rpid;
 int st;

 if (argc < 2) {
  fprintf(stderr, "usage: %s command ...\n", argv[0]);
  return (1);
 }
 if ((pid = fork()) < 0) {
  fprintf(stderr, "fork: %s\n", strerror(errno));
  return (1);
 }
 if (pid == 0) {
  execvp(argv[1], &argv[1]);
  fprintf(stderr, "exec: %s\n", strerror(errno));
  return (1);
 }
 while ((rpid = wait(&st)) > 0 && rpid != pid)
  ;
 if (rpid < 0) {
  fprintf(stderr, "wait: %s\n", strerror(errno));
  return (1);
 }
 printf("status 0x%x\n", st);
 return (0);
}

There is some header-file fumble that prevents me from compiling this with cc -m32, but there are both 32- and 64-bit systems in our environment, so:

Using kernel 3.13.0-59:

Compile it on a 64-bit system, and run
      ./forkexec date
and all is well.

Compile it on a 32-bit system, then, on a 64-bit system, run
    ./forkexec date
and date prints nothing, while forkexec reports exit status 0x8b.

On the other hand, still on the 64-bit system, point it at a 32-bit binary and all is well. e.g.
   ./forkexec ./forkexec
just prints the expected usage: message, so it execed itself properly; no SIGSEGV.

To confound matters further:
-- take out the fork (so the program just calls exec) and all is well
-- run the program under strace -f and the problem vanishes

All this happens under kernel 3.13.0-59 but not 3.13.0-55 (we've put off a few updates).

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-lts-trusty - 3.13.0-61.100~precise1

---------------
linux-lts-trusty (3.13.0-61.100~precise1) precise; urgency=low

  [ Luis Henriques]

  * Re-work previous CVE backports to fix regression
    - LP: #1479093

  [ Upstream Kernel Changes ]

  * Revert "x86/nmi/64: Use DF to avoid userspace RSP confusing nested NMI
    detection"
  * Revert "x86/nmi/64: Reorder nested NMI checks"
  * Revert "x86/nmi/64: Improve nested NMI comments"
  * Revert "x86/nmi/64: Switch stacks on userspace NMI entry"
  * Revert "x86/nmi/64: Remove asm code that saves cr2"
  * Revert "x86/nmi: Enable nested do_nmi handling for 64-bit kernels"
  * Revert "x86/asm/entry/64: Remove pointless jump to irq_return"
  * Revert "x86/asm/entry/64: Remove a redundant jump"
  * Revert "x86/asm/entry/64: Fold the 'test_in_nmi' macro into its only
    user"
  * Revert "x86/asm/entry/64: Always allocate a complete "struct pt_regs"
    on the kernel stack"
  * Revert "x86/asm/64: Open-code register save/restore in
    trace_hardirqs*() thunks"
  * Revert "x86: entry_64.S: fold SAVE_ARGS_IRQ macro into its sole user"
  * Revert "x86: ia32entry.S: fix wrong symbolic constant usage:
    R11->ARGOFFSET"
  * Revert "x86: entry_64.S: delete unused code"
  * Revert "x86, entry: Switch stacks on a paranoid entry from userspace"
  * Revert "x86: Speed up ___preempt_schedule*() by using THUNK helpers"
  * Revert "x86_64, entry: Treat regs->ax the same in fastpath and slowpath
    syscalls"
  * Revert "x86, entry: Only call user_exit if TIF_NOHZ"
  * Revert "x86/debug: Drop several unnecessary CFI annotations"
  * Revert "x86_64, entry: Add missing 'DEFAULT_FRAME 0' entry annotations"
  * x86/asm/entry/64: Fold the 'test_in_nmi' macro into its only user
  * x86/asm/entry/64: Remove a redundant jump
  * x86/nmi: Enable nested do_nmi handling for 64-bit kernels
  * x86/nmi/64: Remove asm code that saves cr2
  * x86/nmi/64: Switch stacks on userspace NMI entry
    - CVE-2015-3290, CVE-2015-5157
  * x86/nmi/64: Improve nested NMI comments
  * x86/nmi/64: Reorder nested NMI checks
  * x86/nmi/64: Use DF to avoid userspace RSP confusing nested NMI
    detection
    - CVE-2015-3291

 -- Luis Henriques <email address hidden> Wed, 29 Jul 2015 12:19:37 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-61.100

---------------
linux (3.13.0-61.100) trusty; urgency=low

  [ Luis Henriques]

  * Re-work previous CVE backports to fix regression
    - LP: #1479093

  [ Upstream Kernel Changes ]

  * Revert "x86/nmi/64: Use DF to avoid userspace RSP confusing nested NMI
    detection"
  * Revert "x86/nmi/64: Reorder nested NMI checks"
  * Revert "x86/nmi/64: Improve nested NMI comments"
  * Revert "x86/nmi/64: Switch stacks on userspace NMI entry"
  * Revert "x86/nmi/64: Remove asm code that saves cr2"
  * Revert "x86/nmi: Enable nested do_nmi handling for 64-bit kernels"
  * Revert "x86/asm/entry/64: Remove pointless jump to irq_return"
  * Revert "x86/asm/entry/64: Remove a redundant jump"
  * Revert "x86/asm/entry/64: Fold the 'test_in_nmi' macro into its only
    user"
  * Revert "x86/asm/entry/64: Always allocate a complete "struct pt_regs"
    on the kernel stack"
  * Revert "x86/asm/64: Open-code register save/restore in
    trace_hardirqs*() thunks"
  * Revert "x86: entry_64.S: fold SAVE_ARGS_IRQ macro into its sole user"
  * Revert "x86: ia32entry.S: fix wrong symbolic constant usage:
    R11->ARGOFFSET"
  * Revert "x86: entry_64.S: delete unused code"
  * Revert "x86, entry: Switch stacks on a paranoid entry from userspace"
  * Revert "x86: Speed up ___preempt_schedule*() by using THUNK helpers"
  * Revert "x86_64, entry: Treat regs->ax the same in fastpath and slowpath
    syscalls"
  * Revert "x86, entry: Only call user_exit if TIF_NOHZ"
  * Revert "x86/debug: Drop several unnecessary CFI annotations"
  * Revert "x86_64, entry: Add missing 'DEFAULT_FRAME 0' entry annotations"
  * x86/asm/entry/64: Fold the 'test_in_nmi' macro into its only user
  * x86/asm/entry/64: Remove a redundant jump
  * x86/nmi: Enable nested do_nmi handling for 64-bit kernels
  * x86/nmi/64: Remove asm code that saves cr2
  * x86/nmi/64: Switch stacks on userspace NMI entry
    - CVE-2015-3290, CVE-2015-5157
  * x86/nmi/64: Improve nested NMI comments
  * x86/nmi/64: Reorder nested NMI checks
  * x86/nmi/64: Use DF to avoid userspace RSP confusing nested NMI
    detection
    - CVE-2015-3291

 -- Luis Henriques <email address hidden> Wed, 29 Jul 2015 10:58:25 +0100

[Cedara (cedara2)]

Thanks for the fix! Kernel 3.13.0-61 solved it just fine.

[Marcel Svitalsky (marcel-svitalsky-gmail)]

Bug #1479111 fixed in the new 3.13.0-61 kernel, thanks a lot!

[Bill Turner, wb4alm (wb4alm)]

This fix also takes care of segfaults in Wine that occurred under kernel 3.13.0.59 (Bug #1479040)
Everything appears to be working just fine under Kernel 3.13.0.61. Thanks everybody!!!

[Norman Wilson (norma7)]

Kernel 3.13.0-61 cures the symptoms I reported as well. Thanks!

[Wayne Schuller (k-wayne)]

Also confirming 3.13.0-61-generic fixes this bug as well.

Dear Ubuntu - don't break Steam ever again! #badmojo :)