sssd/ntpd/postfix + overlayfs startup failure: Could not open file [/var/log/sssd/sssd.log]. Error: [13][Permission denied]

[Graham Leggett]

If an attempt is made to mount an overlay filesystem over the /var/log directory, this causes sssd to refuse to start up.

The startup fails at the point where sssd attempts to write to its logfiles:

sssd: Could not open file [/var/log/sssd/sssd.log]. Error: [13][Permission denied]

sssd is running as root, and should have no problem writing to logfiles. Removing the -f option from sssd causes sshd to not attempt to write to /var/log/ssshd/ssshd.log and sshd startup succeeds.

Running sssd without any flags logs to syslog, and this works correctly, logging to /var/log/syslog on the overlayfs filesystem.

Removing the file /etc/apparmor.d/usr.sbin.sssd causes sssd to start up correctly, logging to the overlayfs /var/log/sssd directory without an issue.

Looks like the apparmour configration for sssd breaks when an overlayfs is present.

[Nish Aravamudan (nacc)]

Hello and thank you for filing this bug report! This does seem like a real issue. Just to confirm, has sssd ever worked with overlays?

[Graham Leggett (minfrin-y)]

I have no idea, all I know is I mounted an overlay disk and managed to completely DoS the machine.

What seems odd is that both sssd and rsyslogd log to /var/log, and both sssd and rsyslogs have an apparmor profile. When /var/log becomes an overlayfs, sssd breaks with permission denied, while rsyslogd keeps working without a problem.

I suspect there is an apparmor profile problem that shows up in sssd but not in rsyslogd.

[Graham Leggett (minfrin-y)]

After an attempt to switch out Trusty for Xenial, this problem now affects more applications.

When /var/log has an overlayfs:

Jun 6 10:34:07 syslog01 sssd: Could not open file [/var/log/sssd/sssd.log]. Error: [13][Permission denied]
Jun 6 10:34:15 syslog01 ntpd[1576]: stat(/var/log/ntpstats/peerstats) failed: Permission denied
Jun 6 10:34:15 syslog01 ntpd[1576]: can't open /var/log/ntpstats/peerstats.20170606: Permission denied
Jun 6 10:34:17 syslog01 ntpd[1576]: stat(/var/log/ntpstats/peerstats) failed: Permission denied

When /var/spool/postfix has an overlayfs:

Jun 5 16:45:17 smtp01 postfix/master[1512]: fatal: remove public/pickup: Permission denied

[Graham Leggett (minfrin-y)]

Zooming in on postfix specifically when /var/spool/postfix is mounted on an overlayfs, postfix goes through the motions of starting but fails silently without logging anything, and /var/log/mail.log remains non-existant.

An attempt to reload postfix complains that postfix isn't running:

root@smtp01:/var/log# postfix reload
postfix: Postfix is running with backwards-compatible default settings
postfix: See http://www.postfix.org/COMPATIBILITY_README.html for details
postfix: To disable backwards compatibility use "postconf compatibility_level=2" and "postfix reload"
postfix/postfix-script: fatal: the Postfix mail system is not running
root@smtp01:/var/log# postfix reload
postfix: Postfix is running with backwards-compatible default settings
postfix: See http://www.postfix.org/COMPATIBILITY_README.html for details
postfix: To disable backwards compatibility use "postconf compatibility_level=2" and "postfix reload"
postfix/postfix-script: fatal: the Postfix mail system is not running

Is overlayfs properly tested and supported on xenial?

[Graham Leggett (minfrin-y)]

If I remove the overlayfs, postfix starts up normally.

[Graham Leggett (minfrin-y)]

Update the package to the linux kernel, as this bug affects multiple services.

[Graham Leggett (minfrin-y)]

I removed apparmor completely, and it made no difference - postfix+overlayfs is still broken without apparmor.

[Graham Leggett (minfrin-y)]

Zooming in to the behaviour of sssd, it appears the permission denied error happens like so:

- A working sssd installation is installed and the daemon started. Logfiles are created in /var/log/sssd, including /var/log/sssd/sssd.log, owned by and exclusively read/writable by root:

root@syslog01:~# ls -al /var/log/sssd
total 12
drwxr-x--- 1 sssd sssd 4096 Jun 6 14:55 .
drwxrwxr-x 1 root syslog 4096 Jun 6 10:45 ..
-rw------- 1 root root 0 Jun 6 14:55 ldap_child.log
-rw------- 1 root root 0 Jun 6 14:55 sssd_LDAP.log
-rw------- 1 root root 0 Jun 6 14:55 sssd.log
-rw------- 1 root root 260 Jun 6 14:56 sssd_nss.log
-rw------- 1 root root 0 Jun 6 14:55 sssd_pam.log
-rw------- 1 root root 0 Jun 6 14:55 sssd_ssh.log
-rw------- 1 root root 0 Jun 6 14:55 sssd_sudo.log

- overlayfs is mounted successfully over /var/log.

- sssd is restarted (manually, or at next boot). sssd cannot open /var/log/sssd/sssd.log despite having permission to do so, with permission denied.

- Manually removing /var/log/sssd/* and restarting sssd causes sssd to start successfully, and the logfiles are recreated successfully with the same mode and user as above.

It seems overlayfs fails at the copy-up step when sssd tries to open existing logfiles that exist in the lowerdir by not yet exist in the upperdir.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.12 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12-rc4

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]