Fatal error : Could not open /run/chrony/chronyd.pid : Permission denied

[Paul Gear]

After an upgrade from 20.04 to 22.04, chrony fails with the error:

Fatal error : Could not open /run/chrony/chronyd.pid : Permission denied

See also an earlier report in https://askubuntu.com/questions/1411005/could-not-open-run-chrony-chronyd-pid-permission-denied/1453066#1453066

The workaround I documented there was:

echo ' @{run}/chrony/{,**} rw,' > /etc/apparmor.d/local/usr.sbin.chronyd
systemctl reload apparmor
systemctl restart chrony

It seems the default chrony apparmor profile should say `@{run}/chrony/{,**} rw,` where it currently says `@{run}/chrony/{,*} rw,`.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: chrony 4.2-2ubuntu2
ProcVersionSignature: Ubuntu 5.15.0-58.64-generic 5.15.74
Uname: Linux 5.15.0-58-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.3
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Thu Feb 2 11:41:41 2023
SourcePackage: chrony
UpgradeStatus: Upgraded to jammy on 2023-02-02 (0 days ago)

[Alex Murray (alexmurray)]

Hmm I am a bit surprised by this - the existing rule would appear to cover /run/chrony/chrony.pid

ie `@{run}/chrony/{,*} rw`, says allow read and write to the directory `/run/chrony/` and any files immediately inside it - of which chrony.pid should be allowed.

Also FWIW I can't reproduce this in a clean 22.04 VM:

root@sec-jammy-amd64:~# apt install chrony
...
root@sec-jammy-amd64:~# aa-status
apparmor module is loaded.
19 profiles are loaded.
19 profiles are in enforce mode.
   /snap/core/14399/usr/lib/snapd/snap-confine
   /snap/core/14399/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/chronyd
   /{,usr/}sbin/dhclient
   lsb_release
   nvidia_modprobe
   nvidia_modprobe//kmod
   snap-update-ns.core
   snap-update-ns.hello-world
   snap.core.hook.configure
   snap.hello-world.env
   snap.hello-world.evil
   snap.hello-world.hello-world
   snap.hello-world.sh
0 profiles are in complain mode.
0 profiles are in kill mode.
0 profiles are in unconfined mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/chronyd (1063)
   /usr/sbin/chronyd (1064)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.
root@sec-jammy-amd64:~# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
     Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-02-02 03:33:19 UTC; 1min 50s ago
       Docs: man:chronyd(8)
             man:chronyc(1)
             man:chrony.conf(5)
    Process: 1054 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
   Main PID: 1063 (chronyd)
      Tasks: 2 (limit: 1120)
     Memory: 1.5M
        CPU: 38ms
     CGroup: /system.slice/chrony.service
             ├─1063 /usr/sbin/chronyd -F 1
             └─1064 /usr/sbin/chronyd -F 1

Feb 02 03:33:19 sec-jammy-amd64 systemd[1]: Starting chrony, an NTP client/server...
Feb 02 03:33:19 sec-jammy-amd64 chronyd[1063]: chronyd version 4.2 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBUG)
Feb 02 03:33:19 sec-jammy-amd64 chronyd[1063]: Using right/UTC timezone to obtain leap second data
Feb 02 03:33:19 sec-jammy-amd64 chronyd[1063]: Loaded seccomp filter (level 1)
Feb 02 03:33:19 sec-jammy-amd64 systemd[1]: Started chrony, an NTP client/server.
Feb 02 03:33:27 sec-jammy-amd64 chronyd[1063]: Selected source 212.243.96.76 (2.ubuntu.pool.ntp.org)
Feb 02 03:33:27 sec-jammy-amd64 chronyd[1063]: System clock TAI offset set to 37 seconds
Feb 02 03:34:34 sec-jammy-amd64 chronyd[1063]: Selected source 185.125.190.56 (ntp.ubuntu.com)

[Christian Ehrhardt  (paelzer)]

I can also not reproduce this in a clean VM, just like Alex.

Also as Alex outlined, there is a rule allowing it:
$ grep 'run.*/chrony' /etc/apparmor.d/usr.sbin.chronyd
  # For /run/chrony to be created
  @{run}/chrony/{,*} rw,
...

hmmm

[Christian Ehrhardt  (paelzer)]

But what might be the difference here is that the reported issues are upgrades, trying for that.

Maybe there is an issue with the old conffile, because back in focal that was slightly different.
If that is still present, then you'd get the issue you see as it changed from /run/chrony.pid to /run/chrony/chrony.pid
$ grep 'run.*/chrony' /etc/apparmor.d/usr.sbin.chronyd
  /{,var/}run/chronyd.pid w,
  /{,var/}run/chrony/{,*} rw,

That changed in:
- https://salsa.debian.org/debian/chrony/-/commit/377e7ca1 Update PID file path
- https://salsa.debian.org/debian/chrony/-/commit/076c5a6a Remove superfluous rule
- https://salsa.debian.org/debian/chrony/-/commit/538fb49b Make use of the @{run} variable

But OTOH I have chrony on my laptop, and there is was no problem when upgrading from Focal to Jammy.
I've had another chrony using system still on Focal which I updated - same result.
No problem, having the new apparmor file after upgrade.

I'm upgrading more systems of mine for comparison ....

[Christian Ehrhardt  (paelzer)]

From IRC here the content of Pauls file - it has all we'd expect:

# grep 'run.*/chrony' /etc/apparmor.d/usr.sbin.chronyd
  # For /run/chrony to be created
  @{run}/chrony/{,*} rw,
  @{run}/chrony-dhcp/{,*} r,
  # Example gpsd socket is outside @{run}/chrony/
  @{run}/chrony.tty{,*}.sock rw,
  @{run}/timemaster/chrony.conf r,

Sadly, even on his system it is now no more reproducible (when removing the rule he added it still works).
Looking for the original denial showed no denial in Pauls logs.
So it seems the "apparmor fix" was a red-herring and just fixed by the restart.

Theory: maybe the profile wasn't reloaded on upgrade?
But then it would show a denial and it didn't.

Also chrony has the common reload in postinst:
$ grep appa /var/lib/dpkg/info/chro*postinst
/var/lib/dpkg/info/chrony.postinst:# Automatically added by dh_apparmor/3.0.3-0ubuntu6
/var/lib/dpkg/info/chrony.postinst: APP_PROFILE="/etc/apparmor.d/usr.sbin.chronyd"
/var/lib/dpkg/info/chrony.postinst: LOCAL_APP_PROFILE="/etc/apparmor.d/local/usr.sbin.chronyd"
/var/lib/dpkg/info/chrony.postinst: apparmor_parser -r -T -W "$APP_PROFILE" || true

[Christian Ehrhardt  (paelzer)]

Upgrading other systems for comparison so far also showed no problems with this.
Nor with file ownership (as asked on askubuntu).

Focal:
ubuntu@ubuntu:~$ sudo ls -laF /run/chrony*
-rw-r--r-- 1 root root 5 Feb 2 07:23 /run/chronyd.pid
/run/chrony:
total 0
drwxr-x--- 2 _chrony _chrony 60 Feb 2 07:44 ./
drwxr-xr-x 33 root root 1040 Feb 2 07:59 ../
srwxr-xr-x 1 _chrony _chrony 0 Feb 2 07:23 chronyd.sock=

Jammy after upgrade:
ubuntu@ubuntu:~$ sudo ls -laF /run/chrony*
/run/chrony:
total 4
drwxr-x--- 2 _chrony _chrony 80 Feb 2 08:25 ./
drwxr-xr-x 34 root root 1060 Feb 2 08:30 ../
-rw-r--r-- 1 root root 6 Feb 2 08:25 chronyd.pid
srwxr-xr-x 1 _chrony _chrony 0 Feb 2 08:25 chronyd.sock=

No change of permissions and no issue (here for me at least)

This is all that came by on do-release-upgrade
ubuntu@ubuntu:~$ journalctl -f | grep chrony
Feb 02 08:01:52 ubuntu sudo[24901]: ubuntu : TTY=pts/3 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/ls -laF /run/chrony /run/chronyd.pid

Feb 02 08:23:27 ubuntu audit[41288]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/chronyd" pid=41288 comm="apparmor_parser"
Feb 02 08:23:27 ubuntu kernel: audit: type=1400 audit(1675326207.963:57): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/chronyd" pid=41288 comm="apparmor_parser"
Feb 02 08:25:54 ubuntu audit[47223]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/chronyd" pid=47223 comm="apparmor_parser"
Feb 02 08:25:54 ubuntu kernel: audit: type=1400 audit(1675326354.637:65): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/chronyd" pid=47223 comm="apparmor_parser"
Feb 02 08:25:57 ubuntu chronyd[7147]: chronyd exiting
Feb 02 08:25:57 ubuntu systemd[1]: Stopping chrony, an NTP client/server...
Feb 02 08:25:57 ubuntu systemd[1]: chrony.service: Deactivated successfully.
Feb 02 08:25:57 ubuntu systemd[1]: Stopped chrony, an NTP client/server.
Feb 02 08:25:57 ubuntu systemd[1]: Starting chrony, an NTP client/server...
Feb 02 08:25:57 ubuntu chronyd[47310]: chronyd version 4.2 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBUG)
Feb 02 08:25:57 ubuntu chronyd[47310]: Frequency 13.180 +/- 0.224 ppm read from /var/lib/chrony/chrony.drift
Feb 02 08:25:57 ubuntu chronyd[47310]: Using right/UTC timezone to obtain leap second data
Feb 02 08:25:57 ubuntu chronyd[47310]: Loaded seccomp filter (level 1)
Feb 02 08:25:57 ubuntu systemd[1]: Started chrony, an NTP client/server.
Feb 02 08:26:04 ubuntu chronyd[47310]: Selected source 194.36.144.87 (0.ubuntu.pool.ntp.org)
Feb 02 08:26:04 ubuntu chronyd[47310]: System clock TAI offset set to 37 seconds
Feb 02 08:27:10 ubuntu chronyd[47310]: Selected source 185.125.190.56 (ntp.ubuntu.com)

And it is working fine after upgrade
ubuntu@ubuntu:~$ systemctl status chrony
● chrony.service - chrony, an NTP client/server
     Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-02-02 08:25:57 UTC; 5min ago
       Docs: ...

[Christian Ehrhardt  (paelzer)]

What do we know so far:

1. The issue in Pauls log was:
Feb 2 11:10:26 dns chronyd[356]: Fatal error : Could not open /run/chrony/chronyd.pid : Permission denied
Feb 2 11:10:26 dns chronyd-starter.sh[354]: Could not open /run/chrony/chronyd.pid : Permission denied

2. It was not apparmor as tuning it back just works for him :-/

3. it was not reproducible in clean VMs, nor on other focal->jammy upgrades nor on Pauls system later on.

:-/

We need to find what is odd for those that are affected by this to fix it.
Keeping it "incomplete"

[Manuel Fombuena (mandolin1)]

I've had a similar error but please disregard if it's not related.

The error was 'Could not open /var/lib/chrony/chrony.drift.tmp : Permission denied'

It turned out that the ownership of /var/lib/chrony was systemd-network:_chrony

Changing it to _chrony:_chrony solved the problem for me.

About the environment. It's a fresh install of Ubuntu 22.04. However it's built using Fenix, a script to build images for Khadas single computer boards (https://github.com/khadas/fenix).

apparmor is not present.

It uses NetworkManager by default and I installed netplan.io. I wonder if that had anything to do with it given the owner became systemd-network.

If there is anything I can provide to help you troubleshoot the issue let me know but since the error isn't exactly the same, it might well be that it's unrelated or particular to my environment.

[Lucas Kanashiro (lucaskanashiro)]

I checked how /var/lib/chrony is handled in the package and I found this in the postinst script:

https://git.launchpad.net/ubuntu/+source/chrony/tree/debian/postinst?h=ubuntu/jammy#n38

It seems that the group and user should be _chrony, not sure how the group was changed to systemd-network. Anyway, this seems to me a different bug from the one originally reported here, if you think this is real problem on the system (not any local config/change) please file another bug so we can track it and move on with a discussion.

[Launchpad Janitor (janitor)]

[Expired for chrony (Ubuntu) because there has been no activity for 60 days.]

[Paul Gear (paulgear)]

I actually encountered this again on an upgrade from Debian buster to bullseye. It gave exactly the same symptoms. The solution this time was to `aa-complain chronyd` followed by `aa-enforce chronyd`. It seems that maybe somehow AppArmor was caching an old profile for `chronyd` somewhere, and this somehow survived multiple reboots on my system.

[Paride Legovini (paride)]

Hello Paul, this bug tracker is for bugs in Ubuntu, while you mention encountering issues with Debian. Please report those in the Debian bugtracker, see:

  https://www.debian.org/Bugs/

and the `reportbug` tool. Thanks!

[Avamander (avamander)]

I encountered the same bug on Ubuntu 22.04.3 LTS, the problem was due to obsolete /etc/apparmor.d/usr.sbin.chronyd.dpkg-dist, making the change as described in the bug description helped.

This system was also upgraded through do-release-upgrade. It might be that some upgrade script is not replacing these profiles with new(er) versions?

[Athos Ribeiro (athos-ribeiro)]

Hi Avamander,

Would you be able to come up with a reproducer (maybe in a VM) where a do-release-upgrade causes the issue consistently?