[17.10 regression] AppArmor ntp denial: Failed name lookup - disconnected path

Bug #1727202 reported by Martin Pitt
100
This bug affects 19 people
Affects Status Importance Assigned to Milestone
ntp (Ubuntu)
Fix Released
Medium
Unassigned
Artful
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Medium
Unassigned
openntpd (Ubuntu)
Incomplete
Low
Unassigned
Artful
Won't Fix
Undecided
Unassigned
Bionic
Won't Fix
Undecided
Unassigned

Bug Description

[Impact]

 * NTP has new isolation features which makes it trigger apparmor issues.
 * Those apparmor issues not only clutter the log and make other things
   less readable, they also prevent ntp from reporting its actual
   messages.
 * Fix is opening the apparmor profile to follow ntp through the
   disconnect by the isolation feature.

[Test Case]

 * This is hard to trigger, but then also not. Which means it is not
   entirely sorted out when it triggers and when not, but the following
   does trigger it in tests of Pitti and also mine (while at the same time
   sometimes it does not - mabye I had other guests or kvm instead of lxd)

 * First install ntp in Artful (or above unless fixed)
   * Install ntp and check demsg for denies
   * Once an issue triggers instead of the error in syslog you'll see the
     apparmor Deny like:
       apparmor="DENIED" operation="sendmsg" info="Failed name lookup -
       disconnected path" error=-13 profile="/usr/sbin/ntpd"
       name="run/systemd/journal/dev-log" pid=5600 comm="ntpd"
       requested_mask="w" denied_mask="w" fsuid=0 ouid=0

[Regression Potential]

 * We are slightly opening up the apparmor profile which is far lower risk
   than adding more constraints. So safe from that POV.

 * OTOH one could think this might be a security issue, but in fact this
   isn't a new suggestion if you take a look at [1] with an ack by Seth of
   the Security Team.

[Other Info]

 * n/a

[1]: https://lists.ubuntu.com/archives/apparmor/2015-May/007858.html

----

Merely installing and starting ntp.service in Ubuntu 17.10 now causes this AppArmor violation:

audit: type=1400 audit(1508915894.215:25): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/ntpd" name="run/systemd/journal/dev-log" pid=5600 comm="ntpd" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

(many times). This hasn't happened in earlier Ubuntu releases yet.

This was spotted by Cockpit's integration tests, as our "ubuntu-stable" image now moved to 17.10 after its release.

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: ntp 1:4.2.8p10+dfsg-5ubuntu3
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu3
Architecture: amd64
Date: Wed Oct 25 03:19:34 2017
SourcePackage: ntp
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Martin Pitt (pitti) wrote :
tags: added: regression-release
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ntp (Ubuntu):
status: New → Confirmed
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Martin, this was missed on triage so far, I ran by today and marked it for the team to look at.
Unfortunately have no cycles atm to fix is up right away.

tags: added: server-next
Changed in ntp (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

As expected I could solve the issue in a test via flags=(attach_disconnected).

Although I had cases where the issue appeared and others where it never showed up - didn't find the difference for that yet. Never the less the fix will help the affected cases and should not break others.

The worst is that I found this to be hiding the actual error message.
In my case where I was seeing it ntp was "actually" complaining about bug 1737998.
But I think atm most ntp errors might look like the apparmor deny which makes this a bit more severe than I thought at first.

Opened an MP for it at [1].

@Martin - do you test Debian as well and if you have apparmor enabled there does it hit there as well? Just to know if you have a bug I could post the fix on

@Martin - since I failed to see when it hits and when not - if you can test from a ppa [2] (or modify the case to have the one line change) it would be nice if you could confirm the fix. If you happen to see why it only happens "sometimes" let me know.

[1]: https://code.launchpad.net/~paelzer/ubuntu/+source/ntp/+git/ntp/+merge/335147
[2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3080

Changed in ntp (Ubuntu Bionic):
importance: Undecided → Medium
Revision history for this message
Richard Laager (rlaager) wrote :

I'm seeing the same errors with NTPsec (a fork of this ntpd, which I have packaged for Debian) on 16.04. The apparmor policy is copied from this ntp package. I'm not able to reproduce the problem at will, but it seems to happen regularly. The proposed change seems to have resolved it.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I was in discussion with Kurt Roeckx on NTPsec before and the NTPsec folks as well but never had the time to continue - thanks you a lot for packaging it up Richard. I subscribed myself to 819806.

Nice to hear that this will fix it for you as well!

Revision history for this message
Martin Pitt (pitti) wrote :

Thanks Christian! Indeed this is rather hard to reproduce locally, but that PR seems to address this. I'll let you know if it doesn't after it lands.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ntp (Ubuntu Artful):
status: New → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ntp - 1:4.2.8p10+dfsg-5ubuntu4

---------------
ntp (1:4.2.8p10+dfsg-5ubuntu4) bionic; urgency=medium

  * debian/apparmor-profile: add attach_disconnected which is needed in some
    cases to let ntp report its log messages (LP: #1727202).

 -- Christian Ehrhardt <email address hidden> Wed, 13 Dec 2017 16:31:30 +0100

Changed in ntp (Ubuntu Bionic):
status: Triaged → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The Bionic fix is about to complete now'ish.
Since this (if triggered) is effectively blocking the messages to appear in the syslog this can hide an arbitrary amount of further issues that the user then has next to no grasp on how to tackle.

So SRU worthy for Artful IMHO.
Adding template soon.

Changed in ntp (Ubuntu Artful):
status: Confirmed → Triaged
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Can you please update the "[Impact]" section of this SRU?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

fwiw, I can see the error in every reboot on my artful vm when it's deployed with maas. I can also therefore verify the fix :)

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Andreas, yes I see it nearly everywhere as well.
Also thanks to spot that I missed to uopdate impact - done.
Discussion on the MP going on ...

description: updated
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Uploaded for the SRU team to evaluate

Changed in ntp (Ubuntu Artful):
status: Triaged → In Progress
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Martin, or anyone else affected,

Accepted ntp into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ntp/1:4.2.8p10+dfsg-5ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ntp (Ubuntu Artful):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-artful
Paul M (speculatrix)
summary: - [17.10 regression] AppArmor denial: Failed name lookup - disconnected
- path
+ [17.10 regression] AppArmor ntp denial: Failed name lookup -
+ disconnected path
Revision history for this message
Gordon Lack (gordon-lack) wrote :

This isn't a 17.10 regression - it's been happening for a few years, e.g.:

    https://bugs.launchpad.net/mos/+bug/1475019

And, FWIW, I added the flags=(attach_disconnected) to the config file yesterday on one of my systems and whereas it does seem to have removed the operation="sendmsg" reports, I still get this at boot time:

Dec 28 14:15:53 parent kernel: [ 24.127330] audit: type=1400 audit(1514470553.526:18): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=1086 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Dec 28 14:15:53 parent kernel: [ 24.127335] audit: type=1400 audit(1514470553.527:19): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/bin/" pid=1086 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Gordon for the extra info.

There are two things in this actually.
1. the disconnected path goes back more release than assumed
   I added tasks since Xenial on the bug here, but even if (for whatever reason) we would decide
   not to push that to X/Z it would not affect the Artful SRu to be stalled once verified to work.

2. the open to its own binary on startup, yes I think I have seen those but this is
   a) a separate issue to be filed for it.
   b) so non severe that no one addressed it so far (might be a nice papercut bug [1])
   I'd assume it already exists as severity low/whishlist, then just the tagging has to be done.

[1]: https://wiki.ubuntu.com/One%20Hundred%20Papercuts

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

At the same time, @Martin are you going to test this with Cockpit or manually against (A-)proposed or should I do so?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

While I see the non-crit "other" issue with opening its own binary I can not confirm the disconnected path issue in a current xenial guest.

Since we knew this appears when trigging the running service to emit an error message I tried to force such an error message. I knew on later releases I could do so by e.g. spawning another virtual interface to bind on by starting a KVM guest (ntp would try to bind on that but fails).
On Xenial I see the error messages without any apparmor related issue.

While I don't know what is different on bug 1475019 (maybe ntp was manually namespaced on that setup) this bug here "as reported" is a regression in 17.10.

Changed in ntp (Ubuntu Xenial):
status: New → Invalid
Changed in ntp (Ubuntu Zesty):
status: New → Invalid
Revision history for this message
Seth Arnold (seth-arnold) wrote :

Why does ntpd try to enumerate the contents of /usr/local/bin/? This in itself isn't so bad but it certainly is curious.

Thanks

Revision history for this message
Martin Pitt (pitti) wrote :

I locally ran Cockpit tests on our current Ubuntu 17.10 image and re-confirm that I got the "disconnected path" error. I then upgraded the ntp package to artful-proposed, and *that* violation is now gone. As others already saw, I now get a test failure on

   apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=5938 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

But this is not a regression from this update, and unrelated. So this SRU is good from my POV. Thanks!

tags: added: verification-done-artful
removed: verification-needed-artful
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Seth, I never checked why it does so but it puzzled me as well, but whatever it is, it is one of those issues that is a) not really critical and b) tries to hide (I spawned X/A guests and containers, no more triggering to take a look at the stack traces of the open - I'm sure it will be back when I have no time to look at it :-) => Heisenbug)

But then again this denial is a different one to the one addressed in the bug here.
So if we - or others - want to continue the discussion on that one we should use another bug.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Martin for verifying!

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Note for comment #22 - I also had B KVM guests and containers now - but it really hides from me today :-)

Revision history for this message
Robie Basak (racb) wrote :

Has anyone actually checked that the new build of ntpd actually still works, please (eg. can sync the time)? If not, please could somebody check that?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Discussion lead a bit off of that, but yes it synced for me in a KVM test.

Revision history for this message
Robie Basak (racb) wrote : Update Released

The verification of the Stable Release Update for ntp has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ntp - 1:4.2.8p10+dfsg-5ubuntu3.1

---------------
ntp (1:4.2.8p10+dfsg-5ubuntu3.1) artful; urgency=medium

  * debian/apparmor-profile: add attach_disconnected which is needed in some
    cases to let ntp report its log messages (LP: #1727202).

 -- Christian Ehrhardt <email address hidden> Mon, 18 Dec 2017 13:19:36 +0100

Changed in ntp (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI - The curiosity of the /usr/local denials will be checked in bug 1741227

Revision history for this message
Martin Pitt (pitti) wrote :

The most plausible explanation for enumerating /usr/local/bin/ is that ntpd has some hooks.d/ mechanism which gets called after syncing the time, and that runs a shell in between. So IMHO this should be allowed.

Revision history for this message
Tim Ritberg (xpert-reactos) wrote :

Problem still present in 18.04

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

@Tim - Could you check the ntp apparmor profile if it has the change that was made in 1:4.2.8p10+dfsg-5ubuntu4 ?
It is a conffile so if depending on your former changes it might have been not updated by default.

Essentially if /etc/apparmor.d/usr.sbin.ntpd has flags=(attach_disconnected) ?

Revision history for this message
Andrew Keynes (andrew-keynes) wrote :

Note that this also appears to affect openntpd in the same fashion, see following for log excerpt of a fresh 18.04 install with the latest openntpd installation:

Nov 23 13:27:34 gbjcdc01 kernel: [1542242.548426] audit: type=1400 audit(1542941854.500:97): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/ntpd" name="run/systemd/journal/dev-log" pid=5693 comm="ntpd" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Andrew, you could try adding:

flags=(attach_disconnected)

to the profile attachment line:

/usr/sbin/ntpd flags=(attach_disconnected) {

And add:

/run/systemd/journal/dev-log w,

to the profile, then run:

apparmor_parser --replace /etc/apparmor.d/usr.sbin.ntpd # or whatever the filename is

See if that lets you get useful logs, any new messages in dmesg or auditd logs, etc.

Thanks

Revision history for this message
Robert Dinse (nanook) wrote : Re: [Bug 1727202] Re: [17.10 regression] AppArmor ntp denial: Failed name lookup - disconnected path
Download full text (4.6 KiB)

      I have since upgraded to 18.10 and I don't even see an apparmor profile
for ntp anymore.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Tue, 27 Nov 2018, Seth Arnold wrote:

> Date: Tue, 27 Nov 2018 01:07:37 -0000
> From: Seth Arnold <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1727202] Re: [17.10 regression] AppArmor ntp denial: Failed name
> lookup - disconnected path
>
> Andrew, you could try adding:
>
> flags=(attach_disconnected)
>
> to the profile attachment line:
>
> /usr/sbin/ntpd flags=(attach_disconnected) {
>
> And add:
>
> /run/systemd/journal/dev-log w,
>
> to the profile, then run:
>
> apparmor_parser --replace /etc/apparmor.d/usr.sbin.ntpd # or whatever
> the filename is
>
> See if that lets you get useful logs, any new messages in dmesg or
> auditd logs, etc.
>
> Thanks
>
> ** Also affects: openntpd (Ubuntu)
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1739943).
> https://bugs.launchpad.net/bugs/1727202
>
> Title:
> [17.10 regression] AppArmor ntp denial: Failed name lookup -
> disconnected path
>
> Status in ntp package in Ubuntu:
> Fix Released
> Status in openntpd package in Ubuntu:
> New
> Status in ntp source package in Xenial:
> Invalid
> Status in openntpd source package in Xenial:
> New
> Status in ntp source package in Zesty:
> Invalid
> Status in openntpd source package in Zesty:
> New
> Status in ntp source package in Artful:
> Fix Released
> Status in openntpd source package in Artful:
> New
> Status in ntp source package in Bionic:
> Fix Released
> Status in openntpd source package in Bionic:
> New
>
> Bug description:
> [Impact]
>
>  * NTP has new isolation features which makes it trigger apparmor issues.
>  * Those apparmor issues not only clutter the log and make other things
>    less readable, they also prevent ntp from reporting its actual
>    messages.
>  * Fix is opening the apparmor profile to follow ntp through the
>    disconnect by the isolation feature.
>
> [Test Case]
>
>  * This is hard to trigger, but then also not. Which means it is not
>    entirely sorted out when it triggers and when not, but the following
>    does trigger it in tests of Pitti and also mine (while at the same time
>    sometimes it does not - mabye I had other guests or kvm instead of lxd)
>
>  * First install ntp in Artful (or above unless fixed)
>    * Install ntp and check demsg for denies
>    * Once an issue triggers instead of the error in syslog you'll see the
>      apparmor Deny like:
>        apparmor="DENIED" operation="sendmsg" info="Failed name lookup -
>        disconnected path" error=-13 profile="/usr/sbin/ntpd"
>        name="run/systemd/journal/dev-log" pid=5600 comm="ntpd"
>        requested_mask="w" denied_mask="w" fsuid=0 ouid=0
>
> [Regression Potential]
>
>  * We are sli...

Read more...

Revision history for this message
Seth Arnold (seth-arnold) wrote :

On Tue, Nov 27, 2018 at 01:22:10AM -0000, Robert Dinse wrote:
> I have since upgraded to 18.10 and I don't even see an apparmor profile
> for ntp anymore.

That's curious. This is in the source package:

# vim:syntax=apparmor
#include <tunables/global>

/usr/sbin/ntpd flags=(attach_disconnected) {
  #include <abstractions/base>
  #include <abstractions/nameservice>

  # conf
  /etc/openntpd/ntpd.conf r,

  # capabilities
  capability kill,
  capability sys_chroot,
  capability setgid,
  capability setuid,
  capability sys_time,
  capability sys_nice,

  /usr/sbin/ntpd mrix,
  /var/lib/openntpd/db/ntpd.drift rw,
  /var/lib/openntpd/run/ntpd.sock rw,

}

It looks like half the change has already been integrated, but not the
systemd-journald socket.

> -_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
> Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
> Knowledgeable human assistance, not telephone trees or script readers.
> See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

Ah this takes me back. :) I learned a huge amount on irc.eskimo.com back
in the day. Belated by two decades, thanks!

Thanks

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Right, the disconnected flag is in the openntpd (usr.sbin.ntpd) profile, but not the journal one:

/run/systemd/journal/dev-log w,

What triggers the journal DENIED error? I see it was in the same DENIED message then had the "disconnected" complaint, but I can't trigger it (as the bug said in the beginning, the error might not happen all the time).

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openntpd (Ubuntu Artful):
status: New → Confirmed
Changed in openntpd (Ubuntu Bionic):
status: New → Confirmed
Changed in openntpd (Ubuntu Xenial):
status: New → Confirmed
Changed in openntpd (Ubuntu Zesty):
status: New → Confirmed
Changed in openntpd (Ubuntu):
status: New → Confirmed
Mathew Hodson (mhodson)
no longer affects: ntp (Ubuntu Xenial)
no longer affects: ntp (Ubuntu Zesty)
no longer affects: openntpd (Ubuntu Xenial)
no longer affects: openntpd (Ubuntu Zesty)
Changed in openntpd (Ubuntu Artful):
status: Confirmed → Won't Fix
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

With chrony taking over for ntpd and the usage of openntpd dropping next to none this really became less and less important over time. It is fixed in ntpd and not affecting chrony.
For openntp it seems to be an issue but we wait for a reply to comment #34 as far as I read through the updates.
Updating tasks to reflect that.

Changed in openntpd (Ubuntu):
status: Confirmed → Incomplete
Changed in openntpd (Ubuntu Bionic):
status: Confirmed → Won't Fix
Changed in openntpd (Ubuntu):
importance: Undecided → Low
tags: removed: server-next
Revision history for this message
Gregory P Smith (gpshead) wrote :

I just diagnosed that openntpd on my 18.04.2 server to be broken (failing to run, the process died after the apparmor denials, no time adjustments ever happening) until I manually applied the changes mentioned in #34.

Neither flags=(attach_disconnected) or "/run/systemd/journal/dev-log w," had been present in my apparmor.d/usr.sbin.ntpd config file. package version 1:6.2p3-1 installed.

Revision history for this message
Gregory P Smith (gpshead) wrote :

(Sadly the bug tracker won't let me change the status from "Won't Fix" to "Confirmed")

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.