isc-dhcp-server fails to renew lease file

[Gabriele Tozzi]

After raring upgrade, the dhcp server fails to renew lease file when it tries to (about every hour).

The syslog says:
dhcpd: Can't create new lease file: Permission denied

It looks like a permission problem, because

# chown -R dhcpd:dhcpd /var/lib/dhcp

the above command temporarily solves the issue, until dhcpd is restarted: at that time, the ownership of the directory and the lease file is set back to root:root.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Bienia (geser)]

The problem is how dhcpd's implements privilege seperation. It doesn't work well with AppArmor and kernel (hard)link protection.

dhcpd expects to be able to write the leases file and create new files in /var/lib/dhcp when rotating the leases file hourly.
As dhcpd is run as user dhcpd, the directory and the files there belonged to dhcpd:dhcpd in the past till it caused a problem with AppArmor (see bug #1028526). As a fix for this the directory and the files belong now root:root and dhcpd can start but not rotate the leases file as user dhcpd anymore (current bug).

Trying to just set dhcpd as owner for /var/lib/dhcp doesn't work as then the kernel hardlink protection triggers when dhcpd tries to hardlink dhcpd.leases (owned by root) to dhcpd.leases~ when rotating the leases file as user dhcpd. Setting dhcpd as the owner of the leases file too, doesn't work either [1] as we are then back where we started.

1: It doesn't work when it belongs dhcpd:dhcpd when the dhcpd get started, but it works when the leases file belongs root when dhcpd gets started and *after* dhcpd got started chowned manually back to dhcpd.

The proper fix is to have dhcpd open the leases file as user dhcpd during start and not root and having /var/lib/dhcp/ and the leases file belong to dhcpd:dhcpd.

[Ubuntu Uefi User (ubuntuuefiuser)]

Will just adding:

sleep 5
chown dhcpd:dhcpd /var/lib/dhcp -R

to the upstart script work?

[Philipp Noack (philipp-noack-b)]

The problem persists in saucy.

[atimonin (atimonin)]

Also have this problem.

Description: Ubuntu 13.10
Release: 13.10
Codename: saucy

[Jasper Knockaert (jasper-f)]

The problem persists in trusty.

[Thiago Martins (martinx)]

I can confirm that this problem persists in Trusty.

[Aleksey Sanin (aleksey-l)]

+1 in Trusty. Sounds like the fix for bug #1028526 needs to be reverted and app_armor fixed instead.

[Ian McMichael (ian-sigma-uk)]

Rather annoyingly, this still seems to be present in 14.04 LTS after all these years! The following seems to fix it (at least until the isc-dhcp-server package gets updated):

- Stop the DHCP server (service isc-dhcp-server stop)
- chown -R dhcpd:dhcpd /var/lib/dhcp
- Edit /etc/init/isc-dhcp-server.conf and remove (or comment out) the following section from near the end:

    # The leases files need to be root:root even when dropping privileges
    [ -e /var/lib/dhcp/dhcpd.leases ] || touch /var/lib/dhcp/dhcpd.leases
    chown root:root /var/lib/dhcp /var/lib/dhcp/dhcpd.leases
    if [ -e /var/lib/dhcp/dhcpd.leases~ ]; then
        chown root:root /var/lib/dhcp/dhcpd.leases~
    fi

- Edit /etc/apparmor.d/local/usr.sbin.dhcpd and add:

# Allow lease file updates
capability dac_override,

- Reload AppArmor profiles (service apparmor reload)
- Restart DHCPd (service isc-dhcp-server start)

Hopefully everything will work now. Why can these changes not be made in the official package?

[Ben Bird (bbird)]

Using the above fix by ian-sigma-uk triggered apparmor bug #1308761, for me.

[Martin Jackson (mhjacks)]

I am running 14.04 as well.

I set my DHCP servers to aa-complain mode. On my secondary (failover) partner, the dhcpd.leases file is owned root:root, even after changing the root:root ownerships to dhcpd:dhcpd in isc-dhcp-server.conf in /etc/init.

I was getting some peculiar address changing behaviors, so I set the split in my failvover conf to 255. This seems to be working, but I find it far from ideal.

[Jamie Strandboge (jdstrand)]

We want to avoid using 'capability dac_override' in the apparmor profile if we can, so the fix in comment #9 is not appropriate for inclusion in Ubuntu.

[Jamie Strandboge (jdstrand)]

As Michael said, this needs a code change to dchpd to open the files correctly.

[atimonin (atimonin)]

I've created a bug-report in www.isc.org:
Your ticket has been assigned an ID of [ISC-Bugs #36978].

[Brian Conry (bconry)]

Can someone please confirm my understanding of this issue?

a) Ubuntu has configured dhcpd to drop root privileges
b) Ubuntu has added logic to dhcpd.conf to force the ownership of dhcpd.leases to root:root
c) Ubuntu is managing the ownership (and permissions?) of the directory in which dhcpd.leases lives, keeping it as root:root

We at ISC are not really clear what we're supposed to do with this.

Thanks,
Brian Conry
ISC Support

[Brian Conry (bconry)]

On my second re-read of this thread I made my spot check and noticed

"
The proper fix is to have dhcpd open the leases file as user dhcpd during start and not root and having /var/lib/dhcp/ and the leases file belong to dhcpd:dhcpd.
"

which implies (correctly) that dhcpd opens dhcpd.leases before dropping privileges, and also implies that maybe the ownership shennanigans are due to dhcpd's failure to open the leases file as the non-privileged user.

This is done because, as things are currently structured, dhcpd initiates the failover relationships (which requires data from the dhcpd.leases file) at the same time as it opens the listening sockets (which requires privileges in the default configuration).

There may also be other information dependencies that I haven't noticed yet.

We're evaluating our options at this point.

It will be helpful to know if the forcing of ownership to root:root of the files and directory are being done as an attempt to cope with dhcpd's behavior or if that is construed as a feature and the desired state.

[Robert Sander (gurubert)]

Workaround from http://ubuntuforums.org/showthread.php?t=2141740&page=2

sudo setfacl -dm u:dhcpd:rwx /var/lib/dhcp
sudo setfacl -m u:dhcpd:rwx /var/lib/dhcp

[DigiAngel (jlay)]

And continuing to see this:

Feb 27 05:43:17 gateway dhcpd: Can't create new lease file: Permission denied
Feb 27 05:43:17 gateway kernel: [ 4703.128481] type=1400 audit(1425040997.320:41): apparmor="DENIED" operation="capable" profile="/usr/sbin/dhcpd" pid=2189 comm="dhcpd" capability=1 capname="dac_override"

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jasper Knockaert (jasper-f)]

Is there a relation with https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1417658?

[Christopher Nighswonger (cnighswonger)]

@bconry: What is the latest on this bug?

[HansLambermont (hans-lambermont)]

This issue persists in 14.04.2 LTS

[Jared Fernandez (jared-fernandez)]

Issue persists in 15.04 Vivid

[Simon McNair (simonmcnair)]

This is still ongoing.
I've tried

- /etc/init/isc-dhcp-server.conf

   # The leases files need to be root:root even when dropping privileges
    [ -e /var/lib/dhcp/dhcpd.leases ] || touch /var/lib/dhcp/dhcpd.leases
    #SM 2015-05-08 chown root:root /var/lib/dhcp /var/lib/dhcp/dhcpd.leases
    chown dhcpd:dhcpd /var/lib/dhcp /var/lib/dhcp/dhcpd.leases
    if [ -e /var/lib/dhcp/dhcpd.leases~ ]; then
    #SM 2015-05-08 chown root:root /var/lib/dhcp/dhcpd.leases~
        echo "Chown DHCPD to fix brokenness"
        chown dhcpd:dhcpd /var/lib/dhcp/dhcpd.leases~
    fi

- /etc/init.d/isc-dhcp-server
...
        start)
                test_config
                log_daemon_msg "Starting $DESC" "$NAME"
                log_daemon_msg "chown dhcpd /var/lib/dhcp/*"
                # SM 2015-08-05 chown dhcpd /var/lib/dhcp/*
                chown dhcpd /var/lib/dhcp/*
...

        restart | force-reload)
                test_config
                $0 stop
                sleep 2
                $0 start
                if [ "$?" != "0" ]; then
                        exit 1
                fi
                #SM 1015-05-08
                chown dhcpd /var/lib/dhcp/*
                ;;
...

-.) Apparmor in to complain mode
mv /etc/apparmor.d/usr.sbin/usr.sbin.dhcpd /etc/apparmor.d/force-complain/

-.)/lib/systemd/system/isc-dhcp-server.service
# SM 2015-05-08 chown root:root /var/lib/dhcp /var/lib/dhcp/dhcpd.leases*; \
    chown dhcpd:dhcpd /var/lib/dhcp /var/lib/dhcp/dhcpd.leases*; \

followed by systemctl daemon-reload

and modifying systemd finally broke it:

systemctl status isc-dhcp-server.service
● isc-dhcp-server.service - ISC DHCP IPv4 server
   Loaded: error (Reason: Invalid argument)
   Active: active (running) since Fri 2015-05-08 09:42:02 BST; 14min ago
     Docs: man:dhcpd(8)
 Main PID: 14253 (dhcpd)
   CGroup: /system.slice/isc-dhcp-server.service
           └─14253 dhcpd -user dhcpd -group dhcpd -f -4 -pf /run/dhcp-server/dhcpd.pid -cf /etc/dhcp/dhcpd.conf eth0...

May 08 09:42:02 here dhcpd[14253]: Server starting service.
May 08 09:42:02 here sh[14253]: Listening on LPF/eth0/00:9c:02:a0:38:67/192.168.1.0/24
May 08 09:42:02 here sh[14253]: Sending on LPF/eth0/00:9c:02:a0:38:67/192.168.1.0/24
May 08 09:42:02 here sh[14253]: Sending on Socket/fallback/fallback-net
May 08 09:42:32 here systemd[1]: [/lib/systemd/system/isc-dhcp-server.service:17] Trailing garbage, ignoring.
May 08 09:42:32 here systemd[1]: isc-dhcp-server.service lacks both ExecStart= and ExecStop= setting. Refusing.
May 08 09:43:04 here systemd[1]: [/lib/systemd/system/isc-dhcp-server.service:17] Trailing garbage, ignoring.
May 08 09:43:04 here systemd[1]: isc-dhcp-server.service lacks both ExecStart= and ExecStop= setting. Refusing.
May 08 09:46:58 here systemd[1]: [/lib/systemd/system/isc-dhcp-server.service:17] Trailing garbage, ignoring.
May 08 09:46:58 here systemd[1]: isc-dhcp-server.service lacks both ExecStart= and ExecStop= setting. Refusing.
root@here:/etc/init.d#

PLEASE can someone tell me how this has persisted since 2013 ?!?!?!

[Ian McMichael (ian-sigma-uk)]

I've given up with my solution in #9 as it did not work. I'm still using 14.04 LTS systems and now employ the following fix instead:

service isc-dhcp-server stop
setfacl -dm u:dhcpd:rwx /var/lib/dhcp
setfacl -m u:dhcpd:rwx /var/lib/dhcp
service isc-dhcp-server start

Note: For this to work you must have acl support on the root filesystem (assuming that's where /var/lib/dhcp lives on your configuration). This is a matter of adding the "acl" option to the mount line in /etc/fstab.

I have been running this for a couple of months now and it seems to do the trick on several servers.

Hope it helps someone?

[Anton Cohen (antoncohen)]

This was fixed in Fedora 18 in 2012, works with SELinux, same version of dhcp-server (4.2.4) as trusty. Maybe a similar method can be used with AppArmor?

https://bugzilla.redhat.com/show_bug.cgi?id=866714
http://pkgs.fedoraproject.org/cgit/dhcp.git/tree/dhcp-paranoia.patch

[Seth Arnold (seth-arnold)]

Nice find Anton; in the little I had inspected the code, I thought it would be solved by moving the db open until after the privileges had been dropped, but that would have significantly complicated the error handling for broken configurations/databases.

This might still require the CAP_CHOWN capability in the AppArmor profile, but at least this would be squashed.

[Linus Nilsson (linusnilsson)]

I'm running Ubuntu Server 14.04.2 LTS and have the same problem.
After I installed the acl-package I'm now using the solution suggested by Ian McMichael and it is working.

[John Center (john-center)]

We're having the same problem running dhcpd under trusty. As a workaround, if we make the file attribute changes in #25, is this the only thing that needs to be done? Can we also change chown root:root to dhcpd:dhcpd for /var/lib/dhcp/* in isc-dhcp-server.conf, too?

Thanks.

    -John

[no!chance (ralf-fehlau)]

This bug exists since 2 years ... I am sure this will never been fixed in ubuntu. ... since you switch to another distro!

[Falk (andreas-mockel)]

Any status on this?

I am pushing a cluster into production with this bug.
Is the "only" way to fix this by using acl'a as Ian McMichael suggested?

service isc-dhcp-server stop
setfacl -dm u:dhcpd:rwx /var/lib/dhcp
setfacl -m u:dhcpd:rwx /var/lib/dhcp
service isc-dhcp-server start

No official fix out there?

--
Regards Falk

[John Center (john-center)]

There are a number of things that need to be addressed with the isc-dhcp-server package. I think I've worked through most of the issues, based on items here & ones I've researched; maybe the maintainer or someone else could review this?

1) /etc/default/isc-dhcp-server needs to be updated to enable several env variables & include one more:

diff -Nru /etc/default/isc-dhcp-server isc-dhcpd-4.2.4/isc-dhcp-server.default
--- /etc/default/isc-dhcp-server 2015-06-19 17:32:49.849591118 -0400
+++ isc-dhcpd-4.2.4/isc-dhcp-server.default 2015-06-19 17:17:36.537576347 -0400
@@ -7,10 +7,13 @@
 #

 # Path to dhcpd's config file (default: /etc/dhcp/dhcpd.conf).
-#DHCPD_CONF=/etc/dhcp/dhcpd.conf
+DHCPD_CONF=/etc/dhcp/dhcpd.conf
+
+# Path to dhcpd's leases file (default: /var/lib/dhcp/dhcpd.leases).
+DHCPD_LEASES=/var/lib/dhcp/dhcpd.leases

 # Path to dhcpd's PID file (default: /var/run/dhcpd.pid).
-#DHCPD_PID=/var/run/dhcpd.pid
+DHCPD_PID=/var/run/dhcp-server/dhcpd.pid

 # Additional options to start dhcpd with.
 # Don't use options -cf or -pf here; use DHCPD_CONF/ DHCPD_PID instead
@@ -18,4 +21,4 @@

2) /etc/init/isc-dhcp-server.conf sources the default isc-dhcp-server file, but does not include the correct env variable to find dhcpd.conf. It should also set the extended file attributes as mentioned earlier. Finally, it should use the env variables defined in the default isc-dhcp-server file when starting the service:

diff -Nru /etc/init/isc-dhcp-server.conf isc-dhcpd-4.2.4/isc-dhcp-server.conf.init
--- /etc/init/isc-dhcp-server.conf 2014-04-03 17:51:15.000000000 -0400
+++ isc-dhcpd-4.2.4/isc-dhcp-server.conf.init 2015-06-19 18:38:04.661654434 -0400
@@ -13,22 +13,17 @@
     fi
     . /etc/default/isc-dhcp-server

- if [ -f /etc/ltsp/dhcpd.conf ]; then
- CONFIG_FILE=/etc/ltsp/dhcpd.conf
- else
- CONFIG_FILE=/etc/dhcp/dhcpd.conf
- fi
- if [ ! -f $CONFIG_FILE ]; then
- echo "$CONFIG_FILE does not exist! - Aborting..."
- echo "Please create and configure $CONFIG_FILE to fix the problem."
+ if [ ! -f $DHCPD_CONF ]; then
+ echo "$DHCPD_CONF does not exist! - Aborting..."
+ echo "Please create and configure $DHCPD_CONF to fix the problem."
         stop
         exit 0
     fi

- if ! dhcpd -user dhcpd -group dhcpd -t -q -4 -cf $CONFIG_FILE > /dev/null 2>&1; then
+ if ! dhcpd -user dhcpd -group dhcpd -t -q -4 -cf $DHCPD_CONF > /dev/null 2>&1; then
         echo "dhcpd self-test failed. Please fix the config file."
         echo "The error was: "
- dhcpd -user dhcpd -group dhcpd -t -4 -cf $CONFIG_FILE
+ dhcpd -user dhcpd -group dhcpd -t -4 -cf $DHCPD_CONF
         stop
         exit 0
     fi
@@ -36,12 +31,6 @@

 respawn
 script
- if [ -f /etc/ltsp/dhcpd.conf ]; then
- CONFIG_FILE=/etc/ltsp/dhcpd.conf
- else
- CONFIG_FILE=/etc/dhcp/dhcpd.conf
- fi
-
     . /etc/default/isc-dhcp-server

     # Allow dhcp server to write lease and pid file as 'dhcpd' user
@@ -50,10 +39,8 @@

     # The leases files need to be root:root even when dropping privileges
     [ -e /var/lib/dhcp/dhcpd.leases ] || touch /var/lib/dhcp/dhcpd.leases
- chown root:root /var/l...

[wdeurholt (wdeurholt)]

I am on Ubuntu 14.04.3 LTS Server.
Bug as described is present.
Solution in #25 works, but still is a workaround.

For the record: Bug was first mentioned 2 years and (neatly) 4 months ago now.
That is amazing.

[Ro (robert-markula)]

Following up on comment #14, this bug (ISC-Bugs #36978) seems to have been fixed in ISC DHCP upstream version 4.3.2 [1]:

" - Enhance the PARANOIA patch to include fchown() the lease file to
     allow it to be manipulated after the server does a chown().
     Thanks to Jiri Popelka at Red Hat for the patch.
     [ISC-Bugs #36978]
"

[1] https://fossies.org/diffs/dhcp/4.3.2_vs_4.3.3/RELNOTES-diff.html

[Skeletor (skeletor99-deactivatedaccount)]

Can this please be fixed in Trusty? Seems pretty essential to have a working DHCP server in an LTS release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package isc-dhcp - 4.3.3-5ubuntu1

---------------
isc-dhcp (4.3.3-5ubuntu1) xenial; urgency=medium

  * Merge from Debian unstable. Remaining changes:
    - Apparmor profiles for dhclient and dhcpd.
    - Create user/group dhcpd.
    - Create /etc/dhcp/ddns-keys/ for DDNS updates.
    - Sanitize environment in dhclient-script.linux.
    - Wait for /etc/resolv.conf to be writable in dhclient-script.linux.
    - If /etc/ltsp/dhcpd.conf exists, use that instead of /etc/dhcp/dhcpd.conf
    - Add an 'option subnet-mask' example to config.
    - Increase the timeout to 300 seconds for dhclient.conf (following the
      default added by dhclient-safer-timeout).
    - Apport hook for isc-dhcp-client and isc-dhcp-server.
    - Upstart jobs for isc-dhcp-server, isc-dhcp-server6, isc-dhcp-relay and
      isc-dhcp-relay6.
    - Separate default file for isc-dhcp-relay6.
    - Drop isc-dhcp-server/new_auth_behavior question from high to medium
    - Remaining Ubuntu patches:
      + dhclient-fix-backoff
      + revert-next-server.
      + multi-ip-addr-per-if.
      + dhclient-safer-timeout.
      + onetry_retry_after_initial_success.
      + dhcp-getifaddrs.patch
      + dhcp-lpf-ib.patch
      + dhcp-improved-xid.patch
      + dhcp-gpxe-cid.patch
      + dhcp-improved-xid-correct-byte-order.patch
      + CVE-2015-8605.patch
  * debian/patches/CVE-2015-8605.patch: refreshed for 4.3.3.
  * debian/patches/system-bind.patch: add -lirs-export to linking libraries
    for dhclient, dhcpd, omshell and dhcrelay since omapip now requires it.
  * debian/isc-dhcp-server*.{upstart,service}: don't chown lease files, since
    the permissions handling in paranoia mode is now done correctly so that
    dhcpd can rotate them. (LP: #1186662)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 13 Jan 2016 15:41:45 -0500

[Jamie Strandboge (jdstrand)]

I came across this bug myself and decided to take a closer look. On trusty, as mentioned, we need the extra PARANOIA patch fro 4.3.3. This will chown the lease file to dhcpd:dhcpd so that afterwards rotation works. I backported a very minimal patch for this. However, the upstart job needed to be adjusted to have this instead:
    ...
    # The leases files need to be root:dhcpd for dropping privileges
    [ -e /var/lib/dhcp/dhcpd.leases ] || touch /var/lib/dhcp/dhcpd.leases
    chown root:dhcpd /var/lib/dhcp /var/lib/dhcp/dhcpd.leases
    chmod 775 /var/lib/dhcp
    chmod 664 /var/lib/dhcp/dhcpd.leases
    ...

'capability chown' needs to be added to the apparmor profile. This allows root to open the file in /var/lib/dhcp without capability dac_override or capability fowner, allows the fchown of the lease file to dhcpd:dhcpd, then allows the dhcpd user to manage the leases and leases~ files. I have test packages in https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages if people want to try them out. If they work for affected users, I'll pursue an SRU to trusty-updates.

I didn't look at xenial very closely, but it doesn't seem to need the root:dhcpd setup. Upstream must have reordered priv dropping and the fchown, etc for this to work. While it would be possible to backport these changes to trusty, I prefer the minimal patch and change to the upstart job in the ppa for a stable release update.

[Jamie Strandboge (jdstrand)]

Since the debdiff in the ppa goes back to the last trusty-security update, here is the debdiff of what is in the ppa.

[Rob Whyte (fudge)]

After installing Xenial server last week I had problems from the get go with isc-dhcp-server and client. As it is my gateway I have had a great deal of trouble. Your suggestions in comment 37 seem to address them, am waiting to see if my next ISP dhcp renewal leaves me disconnected again. Effected packages for me are 4.4.4-5ubuntu7 and 4.4.4-5ubuntu8.

[Jamie Strandboge (jdstrand)]

Rob, the dhcp server issue is bug #1543794 and should be fixed in 4.4.4-5ubuntu9.

[Jamie Strandboge (jdstrand)]

The changes in 4.4.4-5ubuntu9 on xenial are essentially the same as what I put in the ppa for trusty in terms of the directory permissions (note, trusty also updates the PARANOIA patch which xenial already had). Can trusty users comment on if the ppa packages for trusty fixes the issues for you?

[Juri Haberland (haberland)]

Yes, it seems to fix it, but there is a upgrade-problem:

...
Mar 1 21:52:16 server kernel: [177627.390088] init: isc-dhcp-server main process (1218) killed by TERM signal
Mar 1 21:52:20 server dhcpd: Internet Systems Consortium DHCP Server 4.2.4
Mar 1 21:52:20 server dhcpd: Copyright 2004-2012 Internet Systems Consortium.
Mar 1 21:52:20 server dhcpd: All rights reserved.
Mar 1 21:52:20 server dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Mar 1 21:52:20 server dhcpd: Internet Systems Consortium DHCP Server 4.2.4
Mar 1 21:52:20 server dhcpd: Copyright 2004-2012 Internet Systems Consortium.
Mar 1 21:52:20 server dhcpd: All rights reserved.
Mar 1 21:52:20 server dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Mar 1 21:52:20 server kernel: [177631.988612] type=1400 audit(1456865540.867:34): apparmor="DENIED" operation="capable" profile="/usr/sbin/dhcpd" pid=6043 comm="dhcpd" capability=0 capname="chown"
Mar 1 21:52:20 server dhcpd: Can't chown new lease file: Operation not permitted
Mar 1 21:52:20 server kernel: [177631.989335] init: isc-dhcp-server main process (6043) terminated with status 1
Mar 1 21:52:20 server kernel: [177631.989343] init: isc-dhcp-server main process ended, respawning
...

and so on, until:
Mar 1 21:52:21 server kernel: [177632.161456] init: isc-dhcp-server respawning too fast, stopped
Mar 1 21:52:21 server kernel: [177632.248030] type=1400 audit(1456865541.127:38): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/dhcpd" pid=6122 comm="apparmor_parser"
Mar 1 21:52:22 server kernel: [177633.172803] type=1400 audit(1456865542.055:39): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/sbin/dhclient" pid=6186 comm="apparmor_parser"
Mar 1 21:52:22 server kernel: [177633.172970] type=1400 audit(1456865542.055:40): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=6186 comm="apparmor_parser"
Mar 1 21:52:22 server kernel: [177633.173109] type=1400 audit(1456865542.055:41): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=6186 comm="apparmor_parser"

From now on "service isc-dhcp-server start" starts the DHCP server and all seems well. I have currently no idea what's going wrong.

This happend on two servers running Trusty.

[padarjohn (john-meissen)]

Has this been pushed as an update for trusty? I'm running 14.04.4 LTS and have isc-dhcp-server 4.2.4-7ubuntu12.4, and I don't see anything else available from the repos.

[Juri Haberland (haberland)]

No, I just tested a proposed update from Jamie Strandboge's PPA.
See comment #37 and #41.

[Jamie Strandboge (jdstrand)]

For other reasons, I had to move the packages to https://launchpad.net/~jdstrand/+archive/ubuntu/ppa/+packages (my ppa). I just copied the binaries. I need to look into the upgrade problem and also fix the init script to do the same as the upstart job, then I'll regenerate new packages.

[MoD (lluna-nova)]

Is the update problem resolved?

[MoD (lluna-nova)]

Apparently so.

[Rob Traders (rob-traderspit)]

I'm fighting since 14.04 LTS with this lease file issue. The fixes provided in the release of isc-dhcp-server 4.3.3-5ubuntu12 does not fix the behaviour in 16.04 LTS (xenial) for me. The reason is the configuration of apparmaor for the binary /usr/sbin/dhcpd, which prevent the manipulation of the leases file. I tried to fix the capabilities, but can't find a solution. The settings "capability dac_override" has no effect.

Temporary I disabled apparmor interception for isc-dhcpd-server by symlinking to disable folder.
# ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
and restartet the dhcpd. This solves the problem, but it let the dhcpd be unprotected by apparmor.

To fix it finaly, please provide a working /etc/apparmor.d/usr.sbin.dhcpd with the required policy file.

[Sven (5-launmhpad-t)]

Bug still present in Ubuntu 14.04.4 TLS.

[Emmanuel Proust (eproust)]

Bug still present in Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-34-generic x86_64) with version 4.3.3-5ubuntu12.1 of isc-dhcp.

I'll try Rob Traders's fix, but I'm very surprise that Ubuntu leave non fixed such a bug for two years !!
The idea could be to move to another distro with a more efficient fixes delay.

[Emmanuel Proust (eproust)]

Rob Traders's fix does not work for me at all.

For memory it was : "Temporary I disabled apparmor interception for isc-dhcpd-server by symlinking to disable folder.
# ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
and restartet the dhcpd. This solves the problem, but it let the dhcpd be unprotected by apparmor."

[Steve Langasek (vorlon)]

The /etc/apparmor.d/usr.sbin.dhcpd profile in 16.04 includes the line:

  /var/lib/dhcp/dhcpd{,6}.leases* lrw,

So there should be no issue with manipulation of the leases file.

The systemd unit in 16.04 also includes a 'chown' command to ensure the right ownership of the leases file, so there should be no need for any dac override.

Are you somehow running an Ubuntu 16.04 system with upstart as the init instead of systemd? This is what the user in https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1543794/comments/8 appears to have, and this is not supported.

[Christian Reis (kiko)]

I don't think this bug still happens with 16.04 LTS. Can anyone who is seeing it reproduce from a from-scratch install? There could be an issue with upgrades, if people are actually still seeing this.

[Emmanuel Proust (eproust)]

No my system is using systemd.

dpkg -l|grep systemd gives me an answer :

ii libpam-systemd:amd64 229-4ubuntu7 amd64 system and service manager - PAM module
ii libsystemd0:amd64 229-4ubuntu7 amd64 systemd utility library
ii systemd 229-4ubuntu7 amd64 system and service manager
ii systemd-sysv 229-4ubuntu7 amd64 system and service manager - SysV links

dpkg -l|grep upstart doesn't

[Christian Reis (kiko)]

Thanks, understood. Now is this is a fresh install or an upgrade, eproust?

[Emmanuel Proust (eproust)]

Hi kiko,
It a fresh install of Lubuntu 16.04 xenial.
Linux vlubfs1 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[Jared Fernandez (jared-fernandez)]

I'm also still seeing this on 16.04 LTS with systemd. This is an upgrade, not a fresh install.

Aug 26 12:30:52 Host sh[4059]: Can't open /var/lib/dhcp/dhcpd.leases for append.
Aug 26 12:30:52 Host dhcpd[4059]: Copyright 2004-2015 Internet Systems Consortium.
Aug 26 12:30:52 Host kernel: [ 55.961181] audit: type=1400 audit(1472239852.640:16): apparmor="DENIED" operation="capable" profile="/usr/sbin/dhcpd" pid=4059 comm="dhcpd" capability=1 capname="dac_override"
Aug 26 12:30:52 Host dhcpd[4059]: All rights reserved.
Aug 26 12:30:52 Host dhcpd[4059]: For info, please visit https://www.isc.org/software/dhcp/
Aug 26 12:30:52 Host dhcpd[4059]: Can't open /var/lib/dhcp/dhcpd.leases for append.
Aug 26 12:30:52 Host dhcpd[4059]:
Aug 26 12:30:52 Host dhcpd[4059]: If you think you have received this message due to a bug rather
Aug 26 12:30:52 Host dhcpd[4059]: than a configuration issue please read the section on submitting
Aug 26 12:30:52 Host dhcpd[4059]: bugs on either our web page at www.isc.org or in the README file
Aug 26 12:30:52 Host dhcpd[4059]: before submitting a bug. These pages explain the proper
Aug 26 12:30:52 Host dhcpd[4059]: process and the information we find helpful for debugging..
Aug 26 12:30:52 Host dhcpd[4059]:
Aug 26 12:30:52 Host dhcpd[4059]: exiting.
Aug 26 12:30:52 Host systemd[1]: isc-dhcp-server.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 12:30:52 Host systemd[1]: isc-dhcp-server.service: Unit entered failed state.
Aug 26 12:30:52 Host systemd[1]: isc-dhcp-server.service: Failed with result 'exit-code'.

[Jared Fernandez (jared-fernandez)]

One thing I've noticed that may be of interest is that this only occurs for me now during boot up. If manually started with the command "sudo systemctl start isc-dhcp-server.service" it starts up fine with no errors.

[Emsi (trash1-z)]

I got the same on 16.04 after upgrading from 15.10.
It really sucks nobody can fix it for so many years.
Use Slackware guys.

[Emsi (trash1-z)]

For the record:
The proper fix for me was to make sure that /etc/apparmor.d/usr.sbin.dhcpd included
  capability chown,
along:
  capability net_bind_service,
  capability net_raw,
  capability setgid,
  capability setuid,

For some reason it was not there after upgrade.

[Emmanuel Proust (eproust)]

Hi,
Yes it really sucks !
For many of us, if we use Ubuntu it is for the ease of use, so moving to another Linux distro (with probably a lot of other bugs and less ease of use) is not a solution.
If I would accept to lose the ease of use, so I would move to a BSD (OpenBSD for the security) ;-)
Please devs, fix this on Ubuntu !

[Steve Langasek (vorlon)]

On Wed, Sep 07, 2016 at 12:45:04PM -0000, Emsi wrote:
> For the record:
> The proper fix for me was to make sure that /etc/apparmor.d/usr.sbin.dhcpd included
> capability chown,
> along:
> capability net_bind_service,
> capability net_raw,
> capability setgid,
> capability setuid,

> For some reason it was not there after upgrade.

This is a conffile belonging to the isc-dhcp-server package. If this line
was missing, presumably you had a modified /etc/apparmor.d/usr.sbin.dhcpd
file before upgrade, and kept your version on upgrade rather than installing
the version from the new package. Do you have an
/etc/apparmor.d/usr.sbin.dhcpd.dpkg-dist file alongside it? You might want
to see what differences there are between those two files, and replace your
modified profile with the stock one so that you don't run into upgrade
problems in the future.

[Emmanuel Proust (eproust)]

Just checked, I have the five capability lines (below) in the /etc/apparmor.d/usr.sbin.dhcpd of my fresh installed 16.04 and I still have the issue.

  capability chown,
  capability net_bind_service,
  capability net_raw,
  capability setgid,
  capability setuid,

Ready to try any proposed fix...

[Steve Langasek (vorlon)]

eproust, since yours was a fresh install the cause of your problem is unlikely to be the same. But you may want to install the debsums package and check whether any of the files on your system differ from the ones in the package:

$ sudo apt install debsums
$ debsums -e isc-dhcp-server

You may also want to check that your systemd unit is the expected one, by running the following command and attaching the resulting file:

systemctl show isc-dhcp-server > isc-dhcp-server-systemctl.txt

[Emmanuel Proust (eproust)]

# debsums -e isc-dhcp-server
/etc/init.d/isc-dhcp-server OK
/etc/dhcp/dhcpd.conf FAILED
/etc/logcheck/ignore.d.server/isc-dhcp-server OK
/etc/init/isc-dhcp-server6.conf OK
/etc/apparmor.d/usr.sbin.dhcpd OK
/etc/init/isc-dhcp-server.conf OK

[Steve Langasek (vorlon)]

Ok, your 'systemctl show' output matches the default by all relevant measures, and the only modified conffile is the expected one (your dhcpd config). So I'm afraid this brings us no closer to understanding why this fails for some people and not for others.

[Emmanuel Proust (eproust)]

Bad news...

[Seth Arnold (seth-arnold)]

eproust, could you run dmesg | grep DENIED to see if there are AppArmor denials blocking your server?

Thanks

[Emmanuel Proust (eproust)]

Hi set-arnold,
Done a dmesg | grep DENIED
No result.

[Emmanuel Proust (eproust)]

I use the isc-dhcp as a closed/restricted dhcp using the deny unknown-clients; directive.

Please could someone confirm that ALL leases should be listed in /var/lib/dhcp/dhcpd.leases ?

Tomorrow I will test on site my isc-dhcp using :
- the current configuration with clients I will be able to manage,
- an open configuration (removing the unknown-clients; directive.

[Emmanuel Proust (eproust)]

Double checked with several devices connected with static leases.

Nothing else than the following content in my dhcpd.leases file...

cat /var/lib/dhcp/dhcpd.leases

gives :

# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.3.3

server-duid "\000\001\000\001\037@\213\210\000PV\213\320+";

[Dominik (domfi)]

It seems to me, that the leases file is created as root after startup and then the user of the processs changes to dhcp or whatever. I have to approve this yet.
Kind regards.

> Am 07.09.2016 um 23:46 schrieb Steve Langasek <email address hidden>:
>
> Ok, your 'systemctl show' output matches the default by all relevant
> measures, and the only modified conffile is the expected one (your dhcpd
> config). So I'm afraid this brings us no closer to understanding why
> this fails for some people and not for others.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1186662
>
> Title:
> isc-dhcp-server fails to renew lease file
>
> Status in isc-dhcp package in Ubuntu:
> Fix Released
> Status in isc-dhcp source package in Trusty:
> Triaged
>
> Bug description:
> After raring upgrade, the dhcp server fails to renew lease file when
> it tries to (about every hour).
>
> The syslog says:
> dhcpd: Can't create new lease file: Permission denied
>
> It looks like a permission problem, because
>
> # chown -R dhcpd:dhcpd /var/lib/dhcp
>
> the above command temporarily solves the issue, until dhcpd is
> restarted: at that time, the ownership of the directory and the lease
> file is set back to root:root.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1186662/+subscriptions

[Steve Langasek (vorlon)]

On Thu, Sep 29, 2016 at 08:46:59PM -0000, Dominik wrote:

> It seems to me, that the leases file is created as root after startup and
> then the user of the processs changes to dhcp or whatever.

The systemd unit for isc-dhcp-server very explicitly sets the permissions on
/var/lib/dhcp/dhcpd.leases to be writable by the dhcpd process, before it is
ever started.

[Dominik (domfi)]

Sorry. My description was for Xenial and there is no systemd in Xenial (previous LTS, should be fixed there too IMO).

[Dominik (domfi)]

Brr... s/Xenial/Trusty/ Sorry again.

[Emmanuel Proust (eproust)]

Hi all,

Soon 4 years and still no fix for this issue !

I would like to know any suggestion to make this fixed :

- any Canonical contact to explain that the linux community is used to say that Microsoft is very slow to fix bugs...

- any famous community site to inform ubuntu users that such an issue still affects 3 LTS versions (12.04 / 14.04 / 16.04) and it is just a shame.

- ?

I regret a lot to write such comments but I noticed that nothing happens.

[Emmanuel Proust (eproust)]

Hi,

Not seen your message on the launchpad topic.

I would prefer reply on the topic so that the whole subscribers will be
able to read it.

As a summary : Slackware is not an option for me.

I would choose a more professional distro like Opensuse if I had to replace
*buntu. But my first choice would be that *buntu evolve to be become a
professional and reliable distro.
Cheers

Le 7 sept. 2016 9:26 AM, "Emsi" <email address hidden> a écrit :

> I got the same on 16.04 after upgrading from 15.10.
> It really sucks nobody can fix it for so many years.
> Use Slackware guys.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1186662
>
> Title:
> isc-dhcp-server fails to renew lease file
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/
> 1186662/+subscriptions
>

[Paul Henderson (harryhendo)]

Yes, I share your frustration. However, the workaround to change the
permissions of the /var/lib/dhcp directory to dhcpd:dhcpd is working for
me. We shouldn't need a workaround, but at least there is one available.

On 2017-04-30 06:54, Emmanuel Proust wrote:

> Hi all,
>
> Soon 4 years and still no fix for this issue !
>
> I would like to know any suggestion to make this fixed :
>
> - any Canonical contact to explain that the linux community is used to
> say that Microsoft is very slow to fix bugs...
>
> - any famous community site to inform ubuntu users that such an issue
> still affects 3 LTS versions (12.04 / 14.04 / 16.04) and it is just a
> shame.
>
> - ?
>
> I regret a lot to write such comments but I noticed that nothing
> happens.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1186662
>
> Title:
> isc-dhcp-server fails to renew lease file
>
> Status in isc-dhcp package in Ubuntu:
> Fix Released
> Status in isc-dhcp source package in Trusty:
> Triaged
>
> Bug description:
> After raring upgrade, the dhcp server fails to renew lease file when
> it tries to (about every hour).
>
> The syslog says:
> dhcpd: Can't create new lease file: Permission denied
>
> It looks like a permission problem, because
>
> # chown -R dhcpd:dhcpd /var/lib/dhcp
>
> the above command temporarily solves the issue, until dhcpd is
> restarted: at that time, the ownership of the directory and the lease
> file is set back to root:root.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1186662/+subscriptions

[Gaétan QUENTIN (gaetan-quentin)]

ubuntu 17.10 32 bits: the same
Nov 17 22:02:45 nas dhclient[736]: can't create /var/lib/dhcp/dhclient.enp3s0.leases: Read-only file system

drwxr-xr-x 1 root root 44 déc. 13 2016 .
drwxr-xr-x 1 root root 738 juil. 16 18:11 ..
-rw-r--r-- 1 root root 1094 nov. 17 22:02 dhclient.enp3s0.leases

and there is no dhcp user account:

root@nas:/etc# grep -i dhcp /etc/passwd /etc/group
root@nas:/etc#

[Seth Arnold (seth-arnold)]

Gaétan, a read-only filesystem could happen either from a drastic IO error or because the environment is configured to not provide one. I suggest asking on IRC or http://askubuntu.com/ for help debugging your read-only filesystem.

Thanks