isc-dhcp-server fails to start on second & further attempts with 'Can't open /var/lib/dhcp/dhcpd.leases for append'

Bug #1543794 reported by Graham Clinch on 2016-02-09
50
This bug affects 9 people
Affects Status Importance Assigned to Milestone
isc-dhcp (Ubuntu)
High
Jamie Strandboge

Bug Description

During the first start of isc-dhcp-server (using the systemd service unit), a blank dhcpd.leases file is created (presumably owned by root:root). dhcpd starts successfully (and then exits due to no subnet definitions, that's expected).

Attempting to start the service again fails with 'Can't open /var/lib/dhcp/dhcpd.leases for append' (the dhcpd.leases file ownership is now dhcpd.dhcpd), and apparmor logs a denial of the dac_override capability.

Removing the leases file repeats this process (first start works as expected, future ones do not).

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: isc-dhcp-server 4.3.3-5ubuntu4
ProcVersionSignature: Ubuntu 4.4.0-2.16-generic 4.4.0
Uname: Linux 4.4.0-2-generic x86_64
ApportVersion: 2.19.4-0ubuntu2
Architecture: amd64
Date: Tue Feb 9 20:58:12 2016
InstallationDate: Installed on 2016-02-09 (0 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160206)
ProcEnviron:
 LANGUAGE=en_GB:en
 TERM=linux
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: isc-dhcp
UpgradeStatus: No upgrade log present (probably fresh install)

Graham Clinch (g-clinch) wrote :
tags: added: rls-x-incoming
Changed in isc-dhcp (Ubuntu):
importance: Undecided → High
Doug Smythies (dsmythies) wrote :

Try disabling apparmour. That fixed it for me. see also bug 1540672.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in isc-dhcp (Ubuntu):
status: New → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.3.3-5ubuntu8

---------------
isc-dhcp (4.3.3-5ubuntu8) xenial; urgency=medium

  * debian/apparmor/sbin.dhclient: include the openssl abstraction
    (LP: #1543794)

 -- Jamie Strandboge <email address hidden> Thu, 25 Feb 2016 08:32:37 -0600

Changed in isc-dhcp (Ubuntu):
status: Confirmed → Fix Released
Doug Smythies (dsmythies) wrote :

Yes, isc-dhcp - 4.3.3-5ubuntu8 fixes the issue with the dhclient stuff that caused this:

Feb 26 14:05:21 DOUG-64 kernel: [ 17.518717] audit: type=1400 audit(1456524321.220:10): apparmor="DENIED" operation="open" profile="/sbin/dhclient" name="/etc/ssl/openssl.cnf" pid=483 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

However, that is not what this bug report is about. This bug report is about these ones:

Feb 27 07:53:07 DOUG-64 kernel: [ 21.350331] audit: type=1400 audit(1456588387.171:12): apparmor="ALLOWED" operation="capable" profile="/usr/sbin/dhcpd" pid=1248 comm="dhcpd" capability=1 capname="dac_override"
Feb 27 07:53:07 DOUG-64 kernel: [ 21.466959] audit: type=1400 audit(1456588387.287:13): apparmor="ALLOWED" operation="capable" profile="/usr/sbin/dhcpd" pid=1248 comm="dhcpd" capability=3 capname="fowner"

Note: the above are "ALLOWED" because I have it set to complain mode.

which were fixed, but have now returned.
setting back to confirmed.

Changed in isc-dhcp (Ubuntu):
status: Fix Released → Confirmed
Graham Clinch (g-clinch) wrote :

isc-dhcp 4.3.3-5ubuntu8 appears to fix bug #1549736, but not this issue.

Changed in isc-dhcp (Ubuntu):
status: Confirmed → In Progress
assignee: nobody → Jamie Strandboge (jdstrand)
Changed in isc-dhcp (Ubuntu):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.3.3-5ubuntu9

---------------
isc-dhcp (4.3.3-5ubuntu9) xenial; urgency=medium

  * debian/isc-dhcp-server.isc-dhcp-server{,6}.service: adjust permissions
    for most recent PARANOIA commits to avoid granting 'capability
    dac_override' in the AppArmor profile on second and subsequent starts
    (LP: #1543794):
    - /var/lib/dhcp is root:dhcpd and 775
    - /var/lib/dhcp/dhcpd{,6}.leases is root:dhcpd and 664
    - don't chown /run/dhcp-server to dhcpd:dhcpd since the pid file is now
      created as root:root and the pid file was not being created before. If
      this needs to be reintroduced in the future, use root:dhcpd and 775,
      like with the leases file.

 -- Jamie Strandboge <email address hidden> Tue, 01 Mar 2016 08:19:43 -0600

Changed in isc-dhcp (Ubuntu):
status: Fix Committed → Fix Released
Thomas Dreibholz (dreibh) wrote :
Download full text (6.4 KiB)

The problem still exists, for 4.3.3-5ubuntu12.1 (Ubuntu 16.04).

How to reproduce (using /etc/dhcp/dhcpd.conf installed from the package itself, without changes):

1. Starting with a fresh install:

nornetpp@bjoernson:~$ sudo rm -rf /var/lib/dhcp /run/dhcp-server
nornetpp@bjoernson:~$ sudo apt-get install --reinstall isc-dhcp-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/412 kB of archives.
After this operation, 0 B of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 105366 files and directories currently installed.)
Preparing to unpack .../isc-dhcp-server_4.3.3-5ubuntu12.1_amd64.deb ...
Unpacking isc-dhcp-server (4.3.3-5ubuntu12.1) over (4.3.3-5ubuntu12.1) ...
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up isc-dhcp-server (4.3.3-5ubuntu12.1) ...
insserv: warning: script 'screen-cleanup' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for script `screen-cleanup'
insserv: Default-Stop undefined, assuming empty stop runlevel(s) for script `screen-cleanup'
isc-dhcp-server start/running, process 24418

nornetpp@bjoernson:~$ apt-show-versions | grep isc-dhcp-server
isc-dhcp-server:amd64/xenial-updates 4.3.3-5ubuntu12.1 uptodate
isc-dhcp-server:i386 not installed

2. Starting the service fails:

nornetpp@bjoernson:~$ sudo service isc-dhcp-server start
isc-dhcp-server start/running, process 24553
nornetpp@bjoernson:~$ sleep 3
nornetpp@bjoernson:~$ sudo service isc-dhcp-server status
isc-dhcp-server stop/waiting
nornetpp@bjoernson:~$ tail -n20 /var/log/syslog.1
Aug 3 11:23:35 bjoernson dhcpd[24599]: process and the information we find helpful for debugging..
Aug 3 11:23:35 bjoernson dhcpd[24599]:
Aug 3 11:23:35 bjoernson dhcpd[24599]: exiting.
Aug 3 11:23:35 bjoernson dhcpd[24603]: Internet Systems Consortium DHCP Server 4.3.3
Aug 3 11:23:35 bjoernson dhcpd[24603]: Copyright 2004-2015 Internet Systems Consortium.
Aug 3 11:23:35 bjoernson dhcpd[24603]: All rights reserved.
Aug 3 11:23:35 bjoernson dhcpd[24603]: For info, please visit https://www.isc.org/software/dhcp/
Aug 3 11:23:35 bjoernson dhcpd[24604]: Internet Systems Consortium DHCP Server 4.3.3
Aug 3 11:23:35 bjoernson dhcpd[24604]: Copyright 2004-2015 Internet Systems Consortium.
Aug 3 11:23:35 bjoernson dhcpd[24604]: All rights reserved.
Aug 3 11:23:35 bjoernson dhcpd[24604]: For info, please visit https://www.isc.org/software/dhcp/
Aug 3 11:23:35 bjoernson dhcpd[24604]: Can't open /var/lib/dhcp/dhcpd.leases for append.
Aug 3 11:23:35 bjoernson dhcpd[24604]:
Aug 3 11:23:35 bjoernson dhcpd[24604]: If you think you have received this message due to a bug rather
Aug 3 11:23:35 bjoernson dhcpd[24604]: than a configuration issue please read the section on submitting
Aug 3 11:23:35 bjoernson dhcpd[24604]: bugs on either our web page at www.isc.org or in the README file
Aug 3 11:23:35 bjoernson dhcpd[24604]: before submitting a bug. ...

Read more...

Roddie Hasan (roddie) wrote :

I got around it by disabling apparmor for dhcpd:

sudo ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.dhcpd

Roddie Hasan (roddie) wrote :

This is the workaround that I used after upgrading to 16.04:

sudo ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.dhcpd

Steve Langasek (vorlon) wrote :

The output in comment #8 shows that this system is running upstart despite being 16.04. Upstart is not supported as init on 16.04 except during the upgrade. This bug appears to have been fixed for the systemd unit, but not for the upstart job. A simple reboot to systemd should suffice to fix the problem on 16.04.

I have the same issue on Ubuntu 16.04, removing apparmor confs helped.

Rolly Ng (rollyng) wrote :

Issue still exists with fresh installation of 16.04 server.

rolly@head:~$ apt-show-versions | grep isc-dhcp-server
isc-dhcp-server:amd64/xenial-updates 4.3.3-5ubuntu12.6 uptodate

it works with the solution:
sudo ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.dhcpd

I am running dhcpd on infiniband, ipoib on ib0, here is the working output:
rolly@head:~$ sudo service isc-dhcp-server status
[sudo] password for rolly:
● isc-dhcp-server.service - ISC DHCP IPv4 server
   Loaded: loaded (/lib/systemd/system/isc-dhcp-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-01-09 22:05:34 HKT; 19min ago
     Docs: man:dhcpd(8)
 Main PID: 5173 (dhcpd)
    Tasks: 1
   Memory: 9.0M
      CPU: 25ms
   CGroup: /system.slice/isc-dhcp-server.service
           └─5173 dhcpd -user dhcpd -group dhcpd -f -4 -pf /run/dhcp-server/dhcpd.pid -cf /etc/dhcp/dhcpd.conf ib0

Jan 09 22:05:34 head dhcpd[5173]: All rights reserved.
Jan 09 22:05:34 head dhcpd[5173]: For info, please visit https://www.isc.org/software/dhcp/
Jan 09 22:05:34 head dhcpd[5173]: Wrote 0 leases to leases file.
Jan 09 22:05:34 head dhcpd[5173]: Listening on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head sh[5173]: Listening on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head sh[5173]: Sending on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head sh[5173]: Sending on Socket/fallback/fallback-net
Jan 09 22:05:34 head dhcpd[5173]: Sending on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head dhcpd[5173]: Sending on Socket/fallback/fallback-net
Jan 09 22:05:34 head dhcpd[5173]: Server starting service.

Seth Arnold (seth-arnold) wrote :

Rolly, please file a new bug against isc-dhcp and make sure to include the AppArmor DENIED lines from your dmesg or auditd logs. Thanks!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers