unsafe hardlink restrictions deny lease backup

[Anand Kumria]

Occasionally, I see this in my logs:

Feb 4 02:27:07 giskard2 dhcpd[11485]: Can't backup lease database /var/lib/dhcp/dhcpd.leases to /var/lib/dhcp/dhcpd.leases~: Operation not permitted

Feb 4 02:27:07 giskard2 kernel: [237980.192671] audit: type=1702 audit(1517711227.717:14): op=linkat ppid=1 pid=11485 auid=4294967295 uid=111 gid=121 euid=111 suid=111 fsuid=111 egid=121
sgid=121 fsgid=121 tty=(none) ses=4294967295 comm="dhcpd" exe="/usr/sbin/dhcpd" res=0

Feb 4 02:27:07 giskard2 kernel: [237980.192686] audit: type=1302 audit(1517711227.717:15): item=0 name="/var/lib/dhcp/dhcpd.leases" inode=3932557 dev=08:01 mode=0100644 ouid=0 ogid=121
rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

Essentially indicating that the apparmor profile has declined to allow a backup leases file to be created. However, the files does appear to be created. I am unsure why the message is being logged (is the file being created correctly? -- I do not know enough of dhcpd to tell).

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

# dpkg -l | grep dhcp
ii isc-dhcp-client 4.3.3-5ubuntu12.7 amd64 DHCP client for automatically obtaining an IP address
ii isc-dhcp-common 4.3.3-5ubuntu12.7 amd64 common files used by all of the isc-dhcp packages
ii isc-dhcp-server 4.3.3-5ubuntu12.7 amd64 ISC DHCP server for automatic IP address assignment
ii wide-dhcpv6-client 20080615-16 amd64 DHCPv6 client for automatic IPv6 hosts configuration

# dpkg -l | grep apparmor
ii apparmor 2.10.95-0ubuntu2.7 amd64 user-space parser utility for AppArmor
ii apparmor-utils 2.10.95-0ubuntu2.7 amd64 utilities for controlling AppArmor
ii libapparmor-perl 2.10.95-0ubuntu2.7 amd64 AppArmor library Perl bindings
ii libapparmor1:amd64 2.10.95-0ubuntu2.7 amd64 changehat AppArmor library
ii python3-apparmor 2.10.95-0ubuntu2.7 amd64 AppArmor Python3 utility library
ii python3-libapparmor 2.10.95-0ubuntu2.7 amd64 AppArmor library Python3 bindings

# ls -la /var/lib/dhcp
total 16
drwxrwsr-x 2 root dhcpd 4096 Feb 5 01:57 .
drwxr-xr-x 52 root root 4096 Oct 3 2016 ..
-rw-r--r-- 1 root dhcpd 1003 Feb 5 02:27 dhcpd.leases
-rw-r--r-- 1 root dhcpd 1631 Feb 5 01:57 dhcpd.leases~

# find /etc/apparmor
/etc/apparmor
/etc/apparmor/init
/etc/apparmor/init/network-interface-security
/etc/apparmor/init/network-interface-security/sbin.dhclient
/etc/apparmor/init/network-interface-security/usr.sbin.ntpd
/etc/apparmor/severity.db
/etc/apparmor/parser.conf
/etc/apparmor/logprof.conf
/etc/apparmor/subdomain.conf

# find /etc/apparmor.d/
/etc/apparmor.d/
/etc/apparmor.d/usr.sbin.dhcpd
/etc/apparmor.d/sbin.dhclient
/etc/apparmor.d/usr.sbin.rsyslogd
/etc/apparmor.d/usr.sbin.tcpdump
/etc/apparmor.d/usr.sbin.named
/etc/apparmor.d/abstractions
/etc/apparmor.d/abstractions/ubuntu-helpers
/etc/apparmor.d/abstractions/kde
/etc/apparmor.d/abstractions/dbus-session
/etc/apparmor.d/abstractions/nis
/etc/apparmor.d/abstractions/base
/etc/apparmor.d/abstractions/apparmor_api
/etc/apparmor.d/abstractions/apparmor_api/examine
/etc/apparmor.d/abstractions/apparmor_api/introspect
/etc/apparmor.d/abstractions/apparmor_api/change_profile
/etc/apparmor.d/abstractions/apparmor_api/find_mountpoint
/etc/apparmor.d/abstractions/apparmor_api/is_enabled
/etc/apparmor.d/abstractions/nvidia
/etc/apparmor.d/abstractions/ubuntu-browsers
/etc/apparmor.d/abstractions/ubuntu-email
/etc/apparmor.d/abstractions/apache2-common
/etc/apparmor.d/abstractions/private-files
/etc/apparmor.d/abstractions/user-mail
/etc/apparmor.d/abstractions/kerberosclient
/etc/apparmor.d/abstractions/X
/etc/apparmor.d/abstractions/ubuntu-browsers.d
/etc/apparmor.d/abstractions/ubuntu-browsers.d/kde
/etc/apparmor.d/abstractions/ubuntu-browsers.d/mailto
/etc/apparmor.d/abstractions/ubuntu-browsers.d/ubuntu-integration
/etc/apparmor.d/abstractions/ubuntu-browsers.d/plugins-common
/etc/apparmor.d/abstractions/ubuntu-browsers.d/productivity
/etc/apparmor.d/abstractions/ubuntu-browsers.d/multimedia
/etc/apparmor.d/abstractions/ubuntu-browsers.d/user-files
/etc/apparmor.d/abstractions/ubuntu-browsers.d/text-editors
/etc/apparmor.d/abstractions/ubuntu-browsers.d/java
/etc/apparmor.d/abstractions/ubuntu-browsers.d/ubuntu-integration-xul
/etc/apparmor.d/abstractions/enchant
/etc/apparmor.d/abstractions/dovecot-common
/etc/apparmor.d/abstractions/python
/etc/apparmor.d/abstractions/ibus
/etc/apparmor.d/abstractions/ubuntu-unity7-messaging
/etc/apparmor.d/abstractions/ssl_keys
/etc/apparmor.d/abstractions/p11-kit
/etc/apparmor.d/abstractions/mir
/etc/apparmor.d/abstractions/xad
/etc/apparmor.d/abstractions/bash
/etc/apparmor.d/abstractions/ubuntu-console-browsers
/etc/apparmor.d/abstractions/user-write
/etc/apparmor.d/abstractions/postfix-common
/etc/apparmor.d/abstractions/gnome
/etc/apparmor.d/abstractions/ssl_certs
/etc/apparmor.d/abstractions/user-manpages
/etc/apparmor.d/abstractions/consoles
/etc/apparmor.d/abstractions/private-files-strict
/etc/apparmor.d/abstractions/svn-repositories
/etc/apparmor.d/abstractions/authentication
/etc/apparmor.d/abstractions/mysql
/etc/apparmor.d/abstractions/aspell
/etc/apparmor.d/abstractions/ubuntu-feed-readers
/etc/apparmor.d/abstractions/wutmp
/etc/apparmor.d/abstractions/user-download
/etc/apparmor.d/abstractions/winbind
/etc/apparmor.d/abstractions/ubuntu-unity7-base
/etc/apparmor.d/abstractions/ubuntu-unity7-launcher
/etc/apparmor.d/abstractions/dbus
/etc/apparmor.d/abstractions/cups-client
/etc/apparmor.d/abstractions/ubuntu-konsole
/etc/apparmor.d/abstractions/fonts
/etc/apparmor.d/abstractions/mdns
/etc/apparmor.d/abstractions/openssl
/etc/apparmor.d/abstractions/web-data
/etc/apparmor.d/abstractions/user-tmp
/etc/apparmor.d/abstractions/ruby
/etc/apparmor.d/abstractions/dconf
/etc/apparmor.d/abstractions/smbpass
/etc/apparmor.d/abstractions/nameservice
/etc/apparmor.d/abstractions/dbus-strict
/etc/apparmor.d/abstractions/dbus-session-strict
/etc/apparmor.d/abstractions/ubuntu-xterm
/etc/apparmor.d/abstractions/video
/etc/apparmor.d/abstractions/likewise
/etc/apparmor.d/abstractions/xdg-desktop
/etc/apparmor.d/abstractions/ubuntu-bittorrent-clients
/etc/apparmor.d/abstractions/launchpad-integration
/etc/apparmor.d/abstractions/php5
/etc/apparmor.d/abstractions/ubuntu-media-players
/etc/apparmor.d/abstractions/gnupg
/etc/apparmor.d/abstractions/freedesktop.org
/etc/apparmor.d/abstractions/ubuntu-gnome-terminal
/etc/apparmor.d/abstractions/dbus-accessibility
/etc/apparmor.d/abstractions/perl
/etc/apparmor.d/abstractions/orbit2
/etc/apparmor.d/abstractions/audio
/etc/apparmor.d/abstractions/dbus-accessibility-strict
/etc/apparmor.d/abstractions/ubuntu-console-email
/etc/apparmor.d/abstractions/samba
/etc/apparmor.d/abstractions/ldapclient
/etc/apparmor.d/cache
/etc/apparmor.d/cache/.features
/etc/apparmor.d/cache/usr.sbin.dhcpd
/etc/apparmor.d/cache/sbin.dhclient
/etc/apparmor.d/cache/usr.sbin.tcpdump
/etc/apparmor.d/cache/usr.sbin.named
/etc/apparmor.d/cache/usr.sbin.ntpd
/etc/apparmor.d/dhcpd.d
/etc/apparmor.d/tunables
/etc/apparmor.d/tunables/sys
/etc/apparmor.d/tunables/multiarch
/etc/apparmor.d/tunables/securityfs
/etc/apparmor.d/tunables/home
/etc/apparmor.d/tunables/multiarch.d
/etc/apparmor.d/tunables/dovecot
/etc/apparmor.d/tunables/xdg-user-dirs.d
/etc/apparmor.d/tunables/xdg-user-dirs.d/site.local
/etc/apparmor.d/tunables/home.d
/etc/apparmor.d/tunables/home.d/ubuntu
/etc/apparmor.d/tunables/global
/etc/apparmor.d/tunables/proc
/etc/apparmor.d/tunables/ntpd
/etc/apparmor.d/tunables/kernelvars
/etc/apparmor.d/tunables/alias
/etc/apparmor.d/tunables/xdg-user-dirs
/etc/apparmor.d/tunables/apparmorfs
/etc/apparmor.d/usr.sbin.ntpd
/etc/apparmor.d/force-complain
/etc/apparmor.d/disable
/etc/apparmor.d/disable/usr.sbin.rsyslogd
/etc/apparmor.d/local
/etc/apparmor.d/local/usr.sbin.dhcpd
/etc/apparmor.d/local/sbin.dhclient
/etc/apparmor.d/local/usr.sbin.rsyslogd
/etc/apparmor.d/local/README
/etc/apparmor.d/local/usr.sbin.tcpdump
/etc/apparmor.d/local/usr.sbin.named
/etc/apparmor.d/local/usr.sbin.ntpd

# find /etc/apparmor.d/ | grep dhcp | xargs md5sum
accd0d7b6bf25c51c4ee2910ec048b49 /etc/apparmor.d/usr.sbin.dhcpd
d22e7d0dd047de43339e0662cc8e0b0d /etc/apparmor.d/cache/usr.sbin.dhcpd
md5sum: /etc/apparmor.d/dhcpd.d: Is a directory
3f688104e7f181e773b5a50d65510ebc /etc/apparmor.d/local/usr.sbin.dhcpd

# ls -l /etc/apparmor.d/dhcpd.d/
total 0

# cat /etc/apparmor.d/local/usr.sbin.dhcpd
# Site-specific additions and overrides for usr.sbin.dhcpd.
# For more details, please see /etc/apparmor.d/local/README.

# diff -u /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/cache/usr.sbin.dhcpd
Binary files /etc/apparmor.d/usr.sbin.dhcpd and /etc/apparmor.d/cache/usr.sbin.dhcpd differ

# cat /etc/apparmor.d/usr.sbin.dhcpd
# vim:syntax=apparmor
# Last Modified: Mon Jan 25 11:06:45 2016
# Author: Jamie Strandboge <email address hidden>

#include <tunables/global>

/usr/sbin/dhcpd flags=(complain) {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/ssl_keys>

  capability chown,
  capability net_bind_service,
  capability net_raw,
  capability setgid,
  capability setuid,

  network inet raw,
  network packet packet,
  network packet raw,

  @{PROC}/[0-9]*/net/dev r,
  @{PROC}/[0-9]*/net/{dev,if_inet6} r,

  /etc/hosts.allow r,
  /etc/hosts.deny r,

  /etc/dhcp/ r,
  /etc/dhcp/** r,
  /etc/dhcpd{,6}.conf r,
  /etc/dhcpd{,6}_ldap.conf r,

  /usr/sbin/dhcpd mr,

  /var/lib/dhcp/dhcpd{,6}.leases* lrw,
  /var/log/ r,
  /var/log/** rw,
  /{,var/}run/{,dhcp-server/}dhcpd{,6}.pid rw,

  # isc-dhcp-server-ldap
  /etc/ldap/ldap.conf r,

  # LTSP. See:
  # http://www.ltsp.org/~sbalneav/LTSPManual.html
  # https://wiki.edubuntu.org/
  /etc/ltsp/ r,
  /etc/ltsp/** r,
  /etc/dhcpd{,6}-k12ltsp.conf r,
  /etc/dhcpd{,6}.leases* lrw,
  /ltsp/ r,
  /ltsp/** r,

  # Eucalyptus
  /{,var/}run/eucalyptus/net/ r,
  /{,var/}run/eucalyptus/net/** r,
  /{,var/}run/eucalyptus/net/*.pid lrw,
  /{,var/}run/eucalyptus/net/*.leases* lrw,
  /{,var/}run/eucalyptus/net/*.trace lrw,

  # wicd
  /var/lib/wicd/* r,

  # access to bind9 keys for dynamic update
  # It's expected that users will generate one key per zone and have it
  # stored in both /etc/bind9 (for bind to access) and /etc/dhcp/ddns-keys
  # (for dhcpd to access).
  /etc/dhcp/ddns-keys/** r,

  # allow packages to re-use dhcpd and provide their own specific directories
  #include <dhcpd.d>

  # Site-specific additions and overrides. See local/README for details.
  #include <local/usr.sbin.dhcpd>
}

[Anand Kumria (wildfire)]

This problem looks very similiar to https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1543794

[Seth Arnold (seth-arnold)]

These are not AppArmor messages. AppArmor messages clearly say apparmor="DENIED" or apparmor="ALLOWED" or similar.

Audit message 1702 is generated when an application trips a link restriction denial:
https://github.com/torvalds/linux/blob/master/kernel/audit.c#L2254

The "linkat" version of the message comes from:
https://github.com/torvalds/linux/blob/master/fs/namei.c#L968

If you want to enable the unsafe hardlink behaviour you can set a sysctl to do so. (This is a homework problem left for the reader.)

Audit message 1302 reports a filename: https://github.com/torvalds/linux/blob/master/include/uapi/linux/audit.h#L89

Since this is the next audit record after the 1702 message, I believe this is just reporting *which* file was involved in the previous unsafe link behaviour.

Thanks