Comment 2 for bug 1873627

Unfortunately I can't, because I fixed the problem with a workaround and
can't recreate the problem on _this_ server. My workaround was to mount the
new filesystem as /var/log (since the goal was to keep logs from filling up
the root file system), leaving the /var/run symlink on the same filesystem
as /run and now everything works.

If you give me a couple of days I can throw up a new server and see if I
can reproduce the behavior.

Trey Schisser
Waveland Technologies - https://wavelandrcm.com
Director of Security Operations and IT Infrastructure
<email address hidden>
*mobile* 512-496-6660

Never send passwords, regulated data, or confidential information via
unencrypted email.
Please use Signal (https://signal.org) to send short secure messages
(512-496-6660)
Or PGP to send encrypted messages via email (See attached public key)

PGP Public key for <email address hidden>: F056 40E6 AEE2 EB92

-----BEGIN PGP PUBLIC KEY BLOCK-----Version: FlowCrypt 7.6.0 Gmail Encryption
Comment: Seamlessly send and receive encrypted
emailxjMEXmbFHhYJKwYBBAHaRw8BAQdAaaneOd78NEVLKtQROusVcda2zUSTeF2o
NCWvClyafb3NKVRyZXkgU2NoaXNzZXIgPHRzY2hpc3NlckB3YXZlbGFuZHJj
bS5jb20+wncEEBYKAB8FAl5mxR4GCwkHCAMCBBUICgIDFgIBAhkBAhsDAh4B
AAoJEPBWQOau4uuSIcYBAImzI7Dc+63PVDI3OTyL6VoDZOegP1dnC1Jug3wu
1uYBAP4/bZz5Pa1qNgsAuB+HfptfdJvq+EQkbFQv4t7oDwfVAs44BF5mxR4S
CisGAQQBl1UBBQEBB0A5TvjO/keeyllJllXC1fvwKNvADE/T+gNXZJ8EzYVC
GAMBCAfCYQQYFggACQUCXmbFHgIbDAAKCRDwVkDmruLrkkO7AP9CUat2JSbw
nk5fIpKG23eLrZOZ1JGhHQMDYMus/kOXowD/bNVOZ/yoWZ4cWq7gV9/3k3dD
4pxkHA1GaPmQwKr2/gI=
=fqR7
-----END PGP PUBLIC KEY BLOCK-----

On Mon, Apr 20, 2020 at 4:40 PM Seth Arnold <email address hidden>
wrote:

> Running under strace may change the execution environment enough that
> it's not reflective of the actual error, but it's still worth a shot --
> can you pastebin the whole auditd strace logs? That openat() line is
> actually a success -- the error we're looking for will come from the
> audit_set_pid(3) function, which uses netlink, which is an incredibly
> complicated protocol. The error may not look like an error in strace
> output.
>
> Is there any chance the kernel has logged whatever the failure was in
> dmesg output?
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1873627
>
> Title:
> auditd fails after moving /var it a new filesystem and turning
> /var/run into a symlink to /run
>
> Status in audit package in Ubuntu:
> New
>
> Bug description:
> Auditd was working on my system (Ubuntu 18.04LTS, kernel
> 4.15.0-1065-aws) until recently. But after splitting off /var into a
> new filesystem it fails to launch.
>
> running '/sbin/auditd -f' as root indicates a problem writing the pid
> file (no file exists even when it says one does) Post config load command
> output:
> Started dispatcher: /sbin/audispd pid: 16927
> type=DAEMON_START msg=audit(1587280022.692:2019): op=start ver=2.8.2
> format=raw kernel=4.15.0-1065-aws auid=878601141 pid=16925 uid=0 ses=24
> subj=unconfined res=success
> config_manager init complete
> Error setting audit daemon pid (File exists)
> type=DAEMON_ABORT msg=audit(1587280022.692:2020): op=set-pid
> auid=878601141 pid=16925 uid=0 ses=24 subj=unconfined res=failed
> Unable to set audit pid, exiting
> The audit daemon is exiting.
> Error setting audit daemon pid (Permission denied)
>
> /var/run is a symlink to /run
> /var/run permissions are 777 root:root
> /run permissions are 755f root:root
> no /run/auditd.pid and subsiquently no /var/run/auditd.pid exists (even
> though the error incorrectly reports otherwise.
>
> /var/log/audit/audit.log output
> type=DAEMON_START msg=audit(1587278222.942:5617): op=start ver=2.8.2
> format=raw kernel=4.15.0-1065-aws auid=4294967295 pid=7529 uid=0
> ses=4294967295 subj=unconf
> ined res=success
> type=DAEMON_ABORT msg=audit(1587278222.943:5618): op=set-pid
> auid=4294967295 pid=7529 uid=0 ses=4294967295 subj=unconfined res=failed
>
> I have been pulling my hair out over this one. So I ran 'strace
> /sbin/auditd -f' and found the following line in the output.
> "openat(AT_FDCWD, "/var/run/auditd.pid",
> O_WRONLY|O_CREAT|O_TRUNC|O_NOFOLLOW, 0644) = 4"
> I am grasping at straws, but suspect that the O_NOFOLLOW option is
> causing a failure in creating the pid file since /var/run is a symlink. I
> could be wrong but I can't find anything else to suspect.
>
> Since it is best practice to split/var into a separate file system to
> prevent filling the root filesystem in case of an unexpected increase
> in log collection I suspect this is a bug. So either the system needs
> to be able to follow symlinks or an option such as pid_file=[filepath]
> needs to be available in /etc/audit/auditd.conf.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1873627/+subscriptions
>