Calls to /libx32/ld-linux-x32.so.2 hang when using auditd

[Margarita Manterola]

SRU Justification:

Impact: Calls to /libx32/ld-linux-x32.so.2 hang when using auditd
Fix: Upstream, a3c54931199565930d6d84f4c3456f6440aefd41
Testcase: Comment #7

Old Description:

I'm running trusty on a bunch of machines, doing frequent dist-upgrades. My hosts have gcc-multilib installed.

Yesterday, I noticed that initramfs generation was hanging. Today I investigated further and found out that what was hanging were the calls to /libx32/ld-linux-x32.so.2.

This is triggered in initramfs generation because there are at some hooks that incorrectly use copy_exec to copy shell scripts into the initramfs image. In a working machine, when ldd encounters a shell script, it will first call the 64bit linker and since it fails, it will then call the 32bit linker which will also fail.

However, in a machine affected by this bug, the second call will hang forever, preventing new image generation, and package updates in general, when this happens as a trigger for update-initramfs.

Originally I thought this was related to the kernel version, since I was unable to reproduce in a freshly installed machine running -22 and was reproducing it in a machine running -20, but now I'm also reproducing it in a machine running -22, so it must be something else.

I'm sorry I can't provide the exact cause right now, but I think it's worth noting that in some situation there might be a problem, and try to find out which those situations are.

I now have one host running 3.13.0-22-generic, with libc6-x32=2.19-0ubuntu3, where doing ldd /usr/bin/ldd hangs, and another host, with the exact same kernel and libc6-x32 version where doing ldd /usr/bin/ldd produces the expected error message (not a dynamic executable). The main difference is that the first one was installed yesterday and the second one was installed today. Both are dist-upgraded to the latest version of everything.

[Margarita Manterola (marga-9)]

So, one extra comment. All affected machines that I've looked at till now, get fixed when rebooting into the -22 kernel, except one that is a virtual machine (using libvirt). That one is still broken even when running -22.

[Margarita Manterola (marga-9)]

This is happening once again with -24 on physical machines.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Schaller (misch-9)]

I also encountered this issue with -24.
My very forceful workaround was:

sudo killall apt-get dpkg
sudo mv /libx32/ld-linux-x32.so.2 /libx32/ld-linux-x32.so.2.old
sudo dpkg --configure -a
sudo apt-get dist-upgrade
sudo update-initramfs -c -k all # Just to be on the safe side
sudo mv /libx32/ld-linux-x32.so.2.old /libx32/ld-linux-x32.so.2
sudo reboot # To get rid of the hanging and unkillable /libx32/ld-linux-x32.so.2 processes

[Adam Conrad (adconrad)]

Is this on i386 or amd64? I can't reproduce in the slightest here.

[Michael Schaller (misch-9)]

amd64
Adam, is there any more information you would need?
I personally have no clue how this machine ended up in this state.

[Margarita Manterola (marga-9)]

The problem is related to auditd, so you need to have audit rules installed, and sometimes you need to restart auditd to get this to happen.

I reproduced this in a vanilla trusty install by doing the following steps:
 1) Install gcc-multilib (brings in a lot of x32 stuff, just libc6-x32 was not enough)
 2) Add some auditd rules. Apparently which rules you add are not relevant, as long as you add some of them. I'll add an example that might trigger it.
 3) while true; do sudo /etc/init.d/auditd stop && sudo /etc/init.d/auditd start && ldd /bin/busybox; done

it might take a few tries, but the while should hang.

Example rules to add. As I said, which rules you add, it's irrelevant, just some rules are needed:

-a exit,always -F arch=b64 -S open -F dir=/home -F euid!=0 -F a1&2 -F success=1 -C euid!=obj_uid -k permissive_write
-a exit,always -F arch=b64 -S open -F dir=/home -F euid!=0 -F a1&1 -F success=1 -C euid!=obj_uid -k permissive_write
-a exit,always -F arch=b64 -S socket -F success=1 -F a0!=0 -F a0!=1 -F a0!=2 -F a0!=10 -F a0!=16 -F a0!=17 -k funky_socket

When the call hangs, there are two kernel call traces in dmesg. I'll attach the content to this comment.

The faulty function is audit_filter_syscall, and it seems to be related to some data being wrongly sized. More info likely to follow.

[Margarita Manterola (marga-9)]

When you get the necessary behavior for this to hang, it hangs at the brk call. You can see this by running strace -f ldd /bin/busybox (as an example):

[pid 4738] execve("/libx32/ld-linux-x32.so.2", ["/libx32/ld-linux-x32.so.2"], [/* 20 vars */]) = 0
[ Process PID=4738 runs in x32 mode. ]
[pid 4738] brk(0

I'm also attaching the disassembled code and the function that breaks. The specific line is:

0xffffffff810fcdd0 <+144>: mov 0x30(%rbx,%r15,4),%eax

Values of the mentioned registers:

RBX: ffff8800dd446600
R15: 0000000002000000
Value of the faulty result: ffff8800e5446630

Result = RBX + 4*R15 + 0x30

And this corresponds to this memory access in audit_filter_syscall

    if ((e->rule.mask[word] & bit) == bit &&

After much staring at the code and the registers, we believe that the problem is that word is R15 and it's too large. It comes from:

  int word = AUDIT_WORD(ctx->major);

And it's supposed to be the syscall number that is being executed. The assembly code that adds the auditing does not seem to take into account the 32bit nature of this value.

[Philipp Kern (pkern)]

I think this should fix it:

 % diff -Naur /tmp/entry_64.S.orig arch/x86/kernel/entry_64.S
--- /tmp/entry_64.S.orig 2014-05-23 16:12:58.136925093 +0200
+++ arch/x86/kernel/entry_64.S 2014-05-23 16:13:12.229077570 +0200
@@ -699,6 +699,9 @@
  movq %rsi,%rcx /* 4th arg: 2nd syscall arg */
  movq %rdi,%rdx /* 3rd arg: 1st syscall arg */
  movq %rax,%rsi /* 2nd arg: syscall number */
+#if __SYSCALL_MASK != ~0
+ andl $__SYSCALL_MASK,%esi
+#endif
  movl $AUDIT_ARCH_X86_64,%edi /* 1st arg: audit arch */
  call __audit_syscall_entry
  LOAD_ARGS 0 /* reload call-clobbered registers */

The audit entry point expects a non-x32 syscall number in %esi. If the x32 bit is set, it will break horribly by indexing wrongly into the e->rule.mask[word] syscall bitmask. If we reuse the x32 mask (if the kernel is compiled with x32 support), we should be safe. This is already done at various points but syscall_trace_enter called later relies on the bit being set. Otherwise strace will be unaware of the x32 mode and report syscall_$(512+x) instead of the real syscalls.

[Andy Whitcroft (apw)]

Ok I have applied the patch as proposed to a test kernel. Could you test the kernels at the URL below and confirm they do fix things for you:

    http://people.canonical.com/~apw/lp1302605-trusty/

Please report any testing back here.

[Philipp Kern (pkern)]

It seems that my patch was applied in the wrong place. You put it after the 4th arg instead of after %rsi is written (easy to miss given the vastly different assembly argument orders). We actually need to fix up %rsi / 2nd arg syscall number and instead it is now overwritten by the "movq %rax,%rsi" later. So this test kernel does not fix the issue.

[Andy Whitcroft (apw)]

Yeah, that brown paper bag was my fault. Thanks Launchpad for white-space dammaging the patch so I hand applied it in the wrong place. Bah. New kernels for test, with an updated patch included. If you could test and report. Thanks.

[Philipp Kern (pkern)]

Sadly we need a second patch, after which it runs stable in my testing. I'm not into the intrinsic details of x32, but apparently the syscalls come in through both the 64bit and 32bit paths, which I find a bit weird. (At least my first patch made it occur significantly less often.)

Patch attached this time. IS_IA32 is defined like this:

#elif defined CONFIG_IA32_EMULATION
# define IS_IA32 is_compat_task()

process_64.c says this about is_compat_task():

                /* is_compat_task() uses the presence of the x32
                   syscall bit flag to determine compat status */

And indeed:

static inline bool is_compat_task(void)
{
        return is_ia32_task() || is_x32_task();
}

[Philipp Kern (pkern)]

CVE-2014-3917 has been assigned to this issue: http://seclists.org/oss-sec/2014/q2/377

Another proposed patch for this has been posted here: http://article.gmane.org/gmane.linux.kernel/1713179 — This one adds a guard around the array access but also drops syscall auditing for x32 calls.

[Philipp Kern (pkern)]

The following commit landed upstream that at least intends to fix the bug outlined here, even though it does not enable proper auditing for x32.

commit a3c54931199565930d6d84f4c3456f6440aefd41
Author: Andy Lutomirski <email address hidden>
Date: Wed May 28 23:09:58 2014 -0400

    auditsc: audit_krule mask accesses need bounds checking

    Fixes an easy DoS and possible information disclosure.

    This does nothing about the broken state of x32 auditing.

    eparis: If the admin has enabled auditd and has specifically loaded
    audit rules. This bug has been around since before git. Wow...

    Cc: <email address hidden>
    Signed-off-by: Andy Lutomirski <email address hidden>
    Signed-off-by: Eric Paris <email address hidden>
    Signed-off-by: Linus Torvalds <email address hidden>

[Rafael David Tinoco (rafaeldtinoco)]

Attaching patch that is being sent to Kernel Team by e-mail.