Statically linked aide segmentation faults when parsing file acls

[Jeffrey Sloan]

[Impact]

The statically linked version of aide will segmentation fault when it attempts to parse acl data attached to files.

Users will see messages in dmesg similar to:

[ 4101.939249] aide[71672]: segfault at 0 ip 00007f3a132f420b sp 00007fffd6355e28 error 4 in libc-2.31.so[7f3a131f9000+178000]
[ 4101.939257] Code: 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 8b 05 65 ec 0c 00 48 83 ff 10 74 47 48 83 ff 1a 74 49 48 8b 40 60 <48> 8b 10 48 85 d2 75 12 eb 1b 0f 1f 00 48 8b 50 10 48 83 c0 10 48

or on Jammy / Noble (if they build a custom statically linked binary)

[ 2427.555747] aide[36174]: segfault at 1c ip 00007fc5552db6a4 sp 00007fffe963f1e0 error 4 in libnss_systemd.so.2[7fc5552da000+32000]
[ 2427.555754] Code: 84 00 00 00 00 00 85 c0 74 05 8d 68 ff eb d1 b8 b5 ff ff ff eb df f3 0f 1e fa 48 83 ec 08 48 8d 3d 19 49 04 00 e8 dc f5 ff ff <8b> 80 1c 00 00 00 85 c0 0f 95 c0 48 83 c4 08 c3 66 66 2e 0f 1f 84

On Noble, the gdb call stack is:

Thread 3 "aide" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff77a76c0 (LWP 27840)]
0x00007ffff5f4bad5 in _nss_systemd_is_blocked () from /lib/x86_64-linux-gnu/libnss_systemd.so.2
(gdb) bt
#0 0x00007ffff5f4bad5 in _nss_systemd_is_blocked () from /lib/x86_64-linux-gnu/libnss_systemd.so.2
#1 0x00007ffff5f4de58 in _nss_systemd_getpwuid_r () from /lib/x86_64-linux-gnu/libnss_systemd.so.2
#2 0x00000000004fce51 in getpwuid_r ()
#3 0x00000000004fc937 in getpwuid ()
#4 0x00000000004a8ab0 in __acl_to_any_text ()
#5 0x0000000000418dfa in acl2line (line=line@entry=0x7fffe812a350) at ../src/do_md.c:420
#6 0x000000000041bd09 in get_file_attrs (filename=0x83d930 "/var/log/kern_user.log", attr=1216283150910, fs=fs@entry=0x83da40)
    at ../src/gen_list.c:734
#7 0x000000000041406b in file_attrs_worker (arg=<optimized out>) at ../src/db_disk.c:261
#8 0x00000000004ccaf2 in start_thread ()
#9 0x00000000005087dc in clone3 ()

on Focal it is similar, but uses glibc instead of libnss_systemd.so.2 for
getpwuid_r().

Now, getpwuid_r() does a dlopen to load libnss_systemd.so.2, to run _nss_systemd_getpwuid_r(), and this calls _nss_systemd_is_blocked().

Now _nss_systemd_is_blocked() uses Thread Local Storage (TLS):

https://github.com/systemd/systemd/blob/main/src/nss-systemd/nss-systemd.c#L1060

static thread_local unsigned _blocked = 0;

_public_ bool _nss_systemd_is_blocked(void) {
        return _blocked > 0;
}

As we know, Thread Local Storage is allocated at compile time for statically linked binaries. But since the linker had no knowledge that libnss_systemd.so.2 would be loaded at runtime through dlopen, there is no TLS allocated for _blocked. Since the binary is statically linked, there is no allocation for _blocked, and thus, we segmentation fault trying to access it.

There is no fixing this issue unfortunately. All users are suggested to remove aide, and install aide-dynamic instead.

[Testcase]

Start a Focal VM.

1) sudo apt install aide acl
2) sudo cp /var/log/kern.log /var/log/kern_user.log
3) sudo setfacl -m u:12345:r /var/log/kern_user.log
4) cat >> aide-custom.conf << EOF
# Configuration Options
database=file:/var/lib/aide/aide.db.gz
database_out=file:/var/lib/aide/aide.db.new.gz
gzip_dbout = yes
Checksums = sha256+sha512+rmd160+haval+gost+crc32+tiger
database_attrs = Checksums
# Define Rules
PERMS = p+i+u+g+acl+selinux+xattrs
LOG = >
LOG_PLUS = LOG+ANF+ARF
# Define Include Paths
/var/log/ LOG_PLUS
EOF
5) sudo ./aide -i -c aide-custom.conf
Segmentation fault

If you install aide-dynamic instead, it works fine.

$ sudo apt remove aide
$ sudo apt install aide-dynamic

[Where problems could occur]

Users should see no difference moving from aide to aide-dynamic.

aide-dynamic is default from Jammy onward.

There is no real security difference moving to aide-dynamic.

The scary message presented by the focal message can be safely ignored, it is not exactly accurate in the first place.

> This package contains a dynamically linked binary and should only be
> used in exceptional circumstances. To avoid exposure to trojaned
> libraries, it is advised to use one of the statically linked binaries.

[Other info]

Upstream was changed to use dynamic linking by default in:
https://github.com/aide/aide/commit/285e791c0d7c70e3f5e72824562dd27be781c2d6

Upstream bug:
https://github.com/aide/aide/issues/96

Upstream bug (same issue):
https://github.com/htop-dev/htop/issues/503#issuecomment-826007195

Maintainer comment:
https://salsa.debian.org/debian/aide/-/blob/master/debian/aide-common.README.Debian#L318
https://<email address hidden>/msg1827543.html

Upstream Systemd commentary:
https://github.com/systemd/systemd/issues/29337
https://lists.freedesktop.org/archives/systemd-devel/2014-March/017493.html

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Konstantin Kalmykov (kkblkkbl)]

I can tell what I did to mitigate this bug, and It’s worked for me (do it as power user or root):

1. Completely purge installed AIDE:

apt purge aide aide-common
rm -rf /var/log/aide

2. Install AIDE back again:

apt update
apt install aide-common aide-dynamic

3. Then rebuild config:

update-aide.conf
cp -v /var/lib/aide/aide.conf.autogenerated /etc/aide/aide.conf

4. Run init script:

aideinit -f

Then for me it was no errors, and AIDE running well.

[Jeffrey Sloan (jjs-6)]

Thanks - I repeated your recipe step by step, and, sure enough,
running aideinit -f did not cause a segfault.

But after running for over an hour and a half, with zero bytes written to
aide.db.new, it was clearly not going anywhere, so I killed it.

Joe

On Sun, Jul 25, 2021 at 11:40 AM Konstantin Kalmykov <
<email address hidden>> wrote:

> I can tell what I did to mitigate this bug, and It’s worked for me (do
> it as power user or root):
>
> 1. Completely purge installed AIDE:
>
> apt purge aide aide-common
> rm -rf /var/log/aide
>
> 2. Install AIDE back again:
>
> apt update
> apt install aide-common aide-dynamic
>
> 3. Then rebuild config:
>
> update-aide.conf
> cp -v /var/lib/aide/aide.conf.autogenerated /etc/aide/aide.conf
>
> 4. Run init script:
>
> aideinit -f
>
> Then for me it was no errors, and AIDE running well.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1920649
>
> Title:
> ubuntu 20.04 LTS - aide crashes on initialization
>
> Status in aide package in Ubuntu:
> Confirmed
>
> Bug description:
> root@flynn:~# aideinit
> Running aide --init...
> Segmentation fault (core dumped)
> AIDE --init return code 139
>
>
> [Sat Mar 20 18:21:48 2021] aide[6409]: segfault at 0 ip 00007f903f98714b
> sp 00007ffefb573de8 error 4 in libc-2.31.so[7f903f88c000+178000]
> [Sat Mar 20 18:21:48 2021] Code: 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00
> 00 00 f3 0f 1e fa 48 8b 05 25 ad 0c 00 48 83 ff 10 74 47 48 83 ff 1a 74 49
> 48 8b 40 60 <48> 8b 10 48 85 d2 75 12 eb 1b 0f 1f 00 48 8b 50 10 48 83 c0
> 10 48
>
> root@flynn:~# uname -a
> Linux flynn 5.8.0-45-generic #51~20.04.1-Ubuntu SMP Tue Feb 23 13:46:31
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> root@flynn:~#
>
> root@flynn:~# cat /etc/os-release
> NAME="Ubuntu"
> VERSION="20.04.2 LTS (Focal Fossa)"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu 20.04.2 LTS"
> VERSION_ID="20.04"
> HOME_URL="https://www.ubuntu.com/"
> SUPPORT_URL="https://help.ubuntu.com/"
> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> PRIVACY_POLICY_URL="
> https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> VERSION_CODENAME=focal
> UBUNTU_CODENAME=focal
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: aide 0.16.1-1build2
> ProcVersionSignature: Ubuntu 5.8.0-45.51~20.04.1-generic 5.8.18
> Uname: Linux 5.8.0-45-generic x86_64
> ApportVersion: 2.20.11-0ubuntu27.16
> Architecture: amd64
> CasperMD5CheckResult: skip
> Date: Sat Mar 20 18:45:43 2021
> InstallationDate: Installed on 2021-03-20 (0 days ago)
> InstallationMedia: Ubuntu-MATE 20.04.2.0 LTS "Focal Fossa" - Release
> amd64 (20210209.1)
> ProcEnviron:
> TERM=xterm-256color
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: aide
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/aide/+bug/1920649/+subscriptions
>
>

[Konstantin Kalmykov (kkblkkbl)]

>> But after running for over an hour and a half, with zero bytes written to
aide.db.new, it was clearly not going anywhere, so I killed it.

Jeffry, I have officially approved hardware (Dell Vostro laptop, see here Ubuntu HCL: https://ubuntu.com/certified/202006-27974) for my Ubuntu 20.04. And even Ubuntu already was there, prepared by Dell (with OEM 5.10 kernel, not default 5.6 or 5.8):

uname -a
Linux DV5301 5.10.0-1038-oem #40-Ubuntu SMP Fri Jul 16 15:08:30 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

So, first - probably it may be some different with OEM kernel, instead of officially shipped kernel in 20.04 disro ISO.
Second - my laptop has very fast SSD device, but even there AIDE was running around an hour. So If you have, for example, ordinary disk, such as SATA or something like, it seemed to me, it may take long time to wait 'till AIDE finishes check.

[Konstantin Kalmykov (kkblkkbl)]

I was wrong in my previous solutions.

Crashing AIDE is a result of default AIDE config when auditd runnig . When many audit rules are configured, AIDE crashes because audit is running too fast. And AIDE simply can not calculate sums, in case of file changes when calculating. Keep in mind, that auditd reading messages from kernel audit generation mechanisms via syscalls (LSM hooks). Kernel is running really a loooooot of sycalls in every second. The right solutions are:

1. Exclude your audit logs dirs (such as /var/log and/or /var/log/audit) from AIDE rules. Maybe exclude every fast-changing directory from AIDE rules also.
2. Or install and configure AIDE before you configure auditd and (r)syslog rules. But in this case you could take a lot of AIDE messages every check runs.