Warning: The unit file, source configuration file or drop-ins of {apt-news,esm-cache}.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[Nobuto Murata]

[Impact]

systemd may incorrectly mark units as needing a daemon reload, therefore unnecessarily printing the warning that a daemon-reload is needed for the changes to take effect. This is misleading to users, and particularly visible in cases like running apt update, since there is an apt hook (/etc/apt/apt.conf.d/20apt-esm-hook.conf) that starts two systemd services.

[Test Plan]

This bug can be triggered by running systemctl enable/disable on a non-existent unit. After that, calling systemctl start/status/etc. will show a false warning
that the unit changed on disk. Hence, to test, simply:

$ systemctl enable somethingthatdefinitelydoesnotexist.service
$ systemctl status systemd-resolved.service
$ apt update

On an effected system, warnings will be shown when calling both commands after the "enable non-existent unit" step.

[Where problems could occur]

This patch fixes a previous commit by ensuring that the manager->unit_file_state_outdated flag is set *only* when some modifications succeeded, but the overall operation failed. In cases where everything fails, it is not necessary to set the flag.

Since the patch is restricted to this area, any regressions from this patch would most likely also be related to the NeedDaemonReload property of units.

[Other information]

The patch is from upstream: https://github.com/systemd/systemd/pull/32610/commits/d69cba3bfffc7b1e3197e2a34b459db13b1e1cb7#diff-84bbf3361da4139a5620a035ebd24f50a0e4d4ece7952dc2d8b4325e6ebdf6e0L2320

[Original Description]

I recently started seeing the following warning messages when I run `apt update`.

$ sudo apt update
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.
...

apt-news.service for example is in /lib/systemd/system/apt-news.service and it's a static file managed by the package. Does the package maintenance script call systemd related hooks to reload the config whenever the package gets updated?

$ systemctl cat apt-news.service
# /usr/lib/systemd/system/apt-news.service
# APT News is hosted at https://motd.ubuntu.com/aptnews.json and can include
# timely information related to apt updates available to your system.
...

$ dpkg -S /lib/systemd/system/apt-news.service
ubuntu-pro-client: /lib/systemd/system/apt-news.service

ProblemType: BugDistroRelease: Ubuntu 24.04
Package: ubuntu-pro-client 31.1
ProcVersionSignature: Ubuntu 6.6.0-14.14-generic 6.6.3
Uname: Linux 6.6.0-14-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.28.0-0ubuntu1
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Wed Feb 28 13:06:35 2024
InstallationDate: Installed on 2024-01-08 (51 days ago)
InstallationMedia: Ubuntu 24.04 LTS "Noble Numbat" - Daily amd64 (20240104)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>SourcePackage: ubuntu-advantage-tools
UpgradeStatus: No upgrade log present (probably fresh install)
apparmor_logs.txt:

cloud-id.txt-error:
 Failed running command 'cloud-id' [exit(2)]. Message: REDACTED config part /etc/cloud/cloud.cfg.d/99-installer.cfg, insufficient permissions
 REDACTED config part /etc/cloud/cloud.cfg.d/90-installer-network.cfg, insufficient permissions
 REDACTED config part /etc/cloud/cloud.cfg.d/99-installer.cfg, insufficient permissions
 REDACTED config part /etc/cloud/cloud.cfg.d/90-installer-network.cfg, insufficient permissions
livepatch-status.txt-error: Invalid command specified '/snap/bin/canonical-livepatch status'.
uaclient.conf:
 contract_url: https://contracts.canonical.com
 log_level: debug

[Renan Rodrigo (renanrodrigo)]

Hello, Nobuto,

First of all, thanks for reporting this issue.

We did changes to the apt news service file - we added the apparmor profiles and systemd security config there - and no, we didn't reload it by default, which may be causing those warnings.

However, I could not reproduce this behavior. Do you have steps to reproduce it on a fresh system?

I will bring this to the team.

[Nobuto Murata (nobuto)]

It was puzzling indeed, but now I have a reproduction step.

$ sudo apt update
-> no warning

$ sudo apt upgrade
-> to install something to invoke the rsyslog trigger.

Processing triggers for rsyslog (8.2312.0-3ubuntu3) ...
Warning: The unit file, source configuration file or drop-ins of rsyslog.service changed on disk. Run 'systemctl daemon-reload' to reload units.

$ sudo apt update
-> will see the warning.

The warning happens with every systemctl commands so it's not really ubuntu-pro-tools specific issue. However, systemctl warnings are not expected with `apt` commands usually so that's why this could be considered as a surprise. For fixing this properly, the place may not be in pro-tools itself but somewhere else.

[Alberto Contreras (aciba)]

Hello Nobutu. Thanks again for reporting this.

I have been trying to reproduce the error with no success. I tried some combinations of:

- In lxd container with [jammy, noble]
- [pro downgrade / upgrade targeting 31.1]
- pro enable / disable
- [pro downgrade / upgrade targeting 31.1]
- apt updadate
- apt upgrade

Could you please provide more information about it?

Many thanks.

[Nobuto Murata (nobuto)]

I tried to minimize the test case but no luck so far. I will report it back whenever I find something additional.

[Paride Legovini (paride)]

Interestingly this is now happening on my Noble system:

$ sudo apt update
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.

I'm quite sure I didn't manually touch those units.

I first noticed this yesterday 2024-03-03, where apparently nothing relevant happened wrt the u-a-t package. I have 31.1 installed, from the release pocket.

[Paride Legovini (paride)]

Now I remember one relevant thing that happened in the past 48h: I rebooted the affected system.

[Christian Ehrhardt  (paelzer)]

Whoever hits this please help us to spot the difference that happened as we still lack a reproducer.

# check if it has been changed
dpkg --verify ubuntu-advantage-tools

# check if there are drop ins that got added
systemctl cat apt-news.service

@nobotu - was yours really an empty file or did you not copy more than one?

[Nobuto Murata (nobuto)]

> @nobotu - was yours really an empty file or did you not copy more than one?

Are you referring to the `systemctl cat apt-news.service` in the bug description? If so, my apologies. I just pasted the file line of the content on purpose just for confirming the full path of the service. The flie wasn't empty at all and I didn't touch the file manually at all either.

[Nobuto Murata (nobuto)]

Just for completeness.

$ sudo apt update
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Hit:1 http://ftp.riken.jp/Linux/ubuntu noble InRelease
Hit:2 http://ftp.riken.jp/Linux/ubuntu noble-updates InRelease
Hit:3 http://ftp.riken.jp/Linux/ubuntu noble-backports InRelease
Hit:4 http://ftp.riken.jp/Linux/ubuntu noble-proposed InRelease
Hit:5 https://repo.steampowered.com/steam stable InRelease
Hit:6 https://packages.microsoft.com/repos/code stable InRelease
Hit:7 http://security.ubuntu.com/ubuntu noble-security InRelease
Get:8 https://pkgs.tailscale.com/stable/ubuntu noble InRelease
Fetched 6,563 B in 1s (6,699 B/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
67 packages can be upgraded. Run 'apt list --upgradable' to see them.

$ dpkg --verify ubuntu-advantage-tools; echo $?
0

$ apt policy ubuntu-advantage-tools
ubuntu-advantage-tools:
  Installed: 31.1
  Candidate: 31.1
  Version table:
     31.2 100
        100 http://ftp.riken.jp/Linux/ubuntu noble-proposed/main amd64 Packages
        100 http://ftp.riken.jp/Linux/ubuntu noble-proposed/main i386 Packages
 *** 31.1 500
        500 http://ftp.riken.jp/Linux/ubuntu noble/main amd64 Packages
        500 http://ftp.riken.jp/Linux/ubuntu noble/main i386 Packages
        100 /var/lib/dpkg/status

$ systemctl cat apt-news.service
# /usr/lib/systemd/system/apt-news.service
# APT News is hosted at https://motd.ubuntu.com/aptnews.json and can include
# timely information related to apt updates available to your system.
# This service runs in the background during an `apt update` to download the
# latest news and set it to appear in the output of the next `apt upgrade`.
# The script won't do anything if you've run: `pro config set apt_news=false`.
# The script will limit network requests to at most once per 24 hours.
# You can also host your own aptnews.json and configure your system to use it
# with the command:
# `pro config set apt_news_url=https://yourhostname/path/to/aptnews.json`

[Unit]
Description=Update APT News

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /usr/lib/ubuntu-advantage/apt_news.py
AppArmorProfile=ubuntu_pro_apt_news
CapabilityBoundingSet=~CAP_SYS_ADMIN
CapabilityBoundingSet=~CAP_NET_ADMIN
CapabilityBoundingSet=~CAP_NET_BIND_SERVICE
CapabilityBoundingSet=~CAP_SYS_PTRACE
CapabilityBoundingSet=~CAP_NET_RAW
PrivateTmp=true
RestrictAddressFamilies=~AF_NETLINK
RestrictAddressFamilies=~AF_PACKET
# These may break some tests, and should be enabled carefully
#NoNewPrivileges=true
#PrivateDevices=true
#ProtectControlGroups=true
# ProtectHome=true seems to reliably break the GH integration test with a lunar lxd on jammy host
#ProtectHome=true
#ProtectKernelModules=true
#ProtectKernelTunables=true
#ProtectSystem=full
#RestrictSUIDSGID=true
# Unsupported in bionic
# Suggestion from systemd.exec(5) manpage on SystemCallFilter
#SystemCallFilter=@system-service
#SystemCallFilter=~@mount
#SystemC...

[Nobuto Murata (nobuto)]

The list of files modified in the last two hours (if I increase the range to the last 2 days, it lists almost everything).

$ find /etc/systemd /lib/systemd/ -mmin -7200
/etc/systemd/system
/etc/systemd/system/snap-chromium-2768.mount
/etc/systemd/system/snap-hugo-18726.mount
/etc/systemd/system/snap-juju-26548.mount
/etc/systemd/system/sshd-keygen@.service.d
/etc/systemd/system/snap-zoom\x2dclient-225.mount
/etc/systemd/system/snap-hugo-18753.mount
/etc/systemd/system/snap-juju-25751.mount
/etc/systemd/system/graphical.target.wants
/etc/systemd/system/multi-user.target.wants
/etc/systemd/system/multi-user.target.wants/snap-chromium-2768.mount
/etc/systemd/system/multi-user.target.wants/snap-hugo-18726.mount
/etc/systemd/system/multi-user.target.wants/snap-juju-26548.mount
/etc/systemd/system/multi-user.target.wants/snap-zoom\x2dclient-225.mount
/etc/systemd/system/multi-user.target.wants/snap-hugo-18753.mount
/etc/systemd/system/multi-user.target.wants/snap-juju-25751.mount
/etc/systemd/system/multi-user.target.wants/snap-hugo-18706.mount
/etc/systemd/system/snap.juju.fetch-oci.service
/etc/systemd/system/snap-hugo-18706.mount
/etc/systemd/system/snapd.mounts.target.wants
/etc/systemd/system/snapd.mounts.target.wants/snap-chromium-2768.mount
/etc/systemd/system/snapd.mounts.target.wants/snap-hugo-18726.mount
/etc/systemd/system/snapd.mounts.target.wants/snap-juju-26548.mount
/etc/systemd/system/snapd.mounts.target.wants/snap-zoom\x2dclient-225.mount
/etc/systemd/system/snapd.mounts.target.wants/snap-hugo-18753.mount
/etc/systemd/system/snapd.mounts.target.wants/snap-juju-25751.mount
/etc/systemd/system/snapd.mounts.target.wants/snap-hugo-18706.mount
/lib/systemd/system
/lib/systemd/system/tailscaled.service
/lib/systemd/system-generators

[Nobuto Murata (nobuto)]

Hmm, it happened again between those two `apt update`. It might be snapd related.

2024-03-05T10:49:54.513356+09:00 t14 sudo: nobuto : TTY=pts/0 ; PWD=/home/nobuto ; USER=root ; COMMAND=/usr/bin/apt update
2024-03-05T11:00:47.422897+09:00 t14 sudo: nobuto : TTY=pts/0 ; PWD=/home/nobuto ; USER=root ; COMMAND=/usr/bin/apt update

$ uptime
 11:01:51 up 14 min, 1 user, load average: 0.91, 0.90, 0.75

$ find /etc/systemd /lib/systemd -mmin -15
/etc/systemd/system
/etc/systemd/system/snap-go-10535.mount
/etc/systemd/system/multi-user.target.wants
/etc/systemd/system/multi-user.target.wants/snap-go-10535.mount
/etc/systemd/system/snapd.mounts.target.wants
/etc/systemd/system/snapd.mounts.target.wants/snap-go-10535.mount

$ snap refresh --time
timer: 00:00~24:00/4
last: today at 10:53 JST
next: today at 17:07 JST

[Grant Orndorff (orndorffgrant)]

Thank you nobuto! With that I was able to reproduce the issue.

lxc launch ubuntu-daily:noble test
lxc exec test -- apt update # this one works as expected
lxc exec test -- snap install snapd
lxc exec test -- apt update # this one has the warnings in the bug report

assigning this bug to snapd

[Haw Loeung (hloeung)]

Seeing this myself:

| $ sudo apt-get update
| Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
| Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[zeroc (zero-c)]

I get the same warnings after editing 3 files /etc/apt/sources.list.d/

Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.
OK:1 http://security.ubuntu.com/ubuntu noble-security InRelease
Holen:2 https://cli.github.com/packages stable InRelease [3.917 B]
OK:3 https://dl.google.com/linux/chrome/deb stable InRelease
OK:4 https://dl.winehq.org/wine-builds/ubuntu jammy InRelease
OK:5 http://archive.ubuntu.com/ubuntu noble InRelease
OK:6 https://ppa.launchpadcontent.net/oibaf/graphics-drivers/ubuntu noble InRelease
OK:7 https://download.vscodium.com/debs vscodium InRelease
OK:8 https://repo.steampowered.com/steam stable InRelease
Holen:9 https://ppa.launchpadcontent.net/savoury1/gimp/ubuntu jammy InRelease [18,1 kB]
OK:10 http://archive.ubuntu.com/ubuntu noble-updates InRelease
OK:11 http://archive.ubuntu.com/ubuntu noble-backports InRelease
OK:12 https://esm.ubuntu.com/apps/ubuntu noble-apps-security InRelease
OK:13 https://esm.ubuntu.com/apps/ubuntu noble-apps-updates InRelease
OK:14 https://esm.ubuntu.com/infra/ubuntu noble-infra-security InRelease
OK:15 https://esm.ubuntu.com/infra/ubuntu noble-infra-updates InRelease
Holen:16 https://ppa.launchpadcontent.net/savoury1/gimp/ubuntu jammy/main amd64 Packages [26,6 kB]
Holen:17 https://ppa.launchpadcontent.net/savoury1/gimp/ubuntu jammy/main i386 Packages [14,9 kB]
Holen:18 https://ppa.launchpadcontent.net/savoury1/gimp/ubuntu jammy/main Translation-en [14,4 kB]
Es wurden 77,8 kB in 2 s geholt (38,7 kB/s).

[Zygmunt Krynicki (zyga)]

I've reproduced this and collected forkstat logs from installation of snapd snap on an otherwise pristine "noble" system. I think what is going on is that systemd stays in a mode where it knows that units on disk have changed vs units in memory and will print the warning until re-loaded. The fact that apt hooks fiddle with systemd units is sufficient for printing the warning:

apt update causes this thing to execute:

10:37:56 exec 3523 sh -c -- [ ! -e /run/systemd/system ] || [ $(id -u) -ne 0 ] || systemctl start --no-block apt-news.service esm-cache.service || true

This is enough for the warning.

The remaining question is where in the installation of snapd do we modify units after last daemon-reload. I'm focusing on that aspect now.

[Zygmunt Krynicki (zyga)]

Snapd touches neither apt-news.service nor esm-cache.service.

On my system the only mention of esm-cache.service is in uaclient/actions.py:

zyga@ciri:/$ grep -FR esm-cache.service usr/ 2>/dev/null
usr/lib/python3/dist-packages/uaclient/actions.py: "esm-cache.service",

I've increased systemd logging to debug to see what is replacing the service but I cannot find any evidence of that in the logs.

[Zygmunt Krynicki (zyga)]

Removing ubuntu-pro-client silences this, so that installation of snapd snap no longer causes any side-effects. While I can see that installation of snapd has some impact on ubuntu-pro-client, I cannot yet understand how.

[Andreas Hasenack (ahasenack)]

Check the postinst script of the binary packages produced by src:ubuntu-advantage-tools. The binary packages that install systemd units must call daemon-reload at some point after the new unit file was installed.

[Nobuto Murata (nobuto)]

It's not the apt-news nor esm-cache service that was modified.

It looks like systemd warns about daemon-reload in any cases if any of the systemd unit files are modified and daemon-reload wasn't called after that.
https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2055239/comments/12

[Andreas Hasenack (ahasenack)]

> It's not the apt-news nor esm-cache service that was modified.

> It looks like systemd warns about daemon-reload in any cases if any of the systemd unit files are
> modified and daemon-reload wasn't called after that.

I understand, but in comment #14 the warning is very specific about the unit files that changed: apt-news.service and esm-cache.service

Could it be that something else installed an override config for those units elsewhere (/run, or /etc), and then didn't issue the daemon-reload?

Could we get an "systemctl cat apt-news.service esm-cache.service" output after this warning? It will say which files exactly are being considered, if it's just /lib/systemd/system/{apt-news,esm-cache}.service or other config snippets.

[Zygmunt Krynicki (zyga)]

With a closer look I ended up running this loop while looking at systemd debug logs:

sudo snap remove --purge snapd && sudo systemctl daemon-reload && sudo systemctl restart snapd && snap version && sudo apt update && echo "ALOHA: installing snapd" | systemd-cat && sudo snap install snapd && echo "ALOHA: done installing snapd" | systemd-cat

This causes the following log file to show up:

mar 13 13:02:52 ciri systemd[1]: Looking for unit files in (higher priority first):
mar 13 13:02:52 ciri systemd[1]: /etc/systemd/system.control
mar 13 13:02:52 ciri systemd[1]: /run/systemd/system.control
mar 13 13:02:52 ciri systemd[1]: /run/systemd/transient
mar 13 13:02:52 ciri systemd[1]: /run/systemd/generator.early
mar 13 13:02:52 ciri systemd[1]: /etc/systemd/system
mar 13 13:02:52 ciri systemd[1]: /etc/systemd/system.attached
mar 13 13:02:52 ciri systemd[1]: /run/systemd/system
mar 13 13:02:52 ciri systemd[1]: /run/systemd/system.attached
mar 13 13:02:52 ciri systemd[1]: /run/systemd/generator
mar 13 13:02:52 ciri systemd[1]: /usr/local/lib/systemd/system
mar 13 13:02:52 ciri systemd[1]: /usr/lib/systemd/system
mar 13 13:02:52 ciri systemd[1]: /run/systemd/generator.late
mar 13 13:02:52 ciri systemd[1]: Modification times have changed, need to update cache.

The message at the bottom of the log comes from systemd's src/basic/unit-file.c

bool lookup_paths_timestamp_hash_same(const LookupPaths *lp, uint64_t timestamp_hash, uint64_t *ret_new) {
        struct siphash state;

        siphash24_init(&state, HASH_KEY.bytes);

        STRV_FOREACH(dir, lp->search_path) {
                struct stat st;

                if (lookup_paths_mtime_exclude(lp, *dir))
                        continue;

                /* Determine the latest lookup path modification time */
                if (stat(*dir, &st) < 0) {
                        if (errno == ENOENT)
                                continue;

                        log_debug_errno(errno, "Failed to stat %s, ignoring: %m", *dir);
                        continue;
                }

                siphash24_compress_usec_t(timespec_load(&st.st_mtim), &state);
        }

        uint64_t updated = siphash24_finalize(&state);
        if (ret_new)
                *ret_new = updated;
        if (updated != timestamp_hash)
                log_debug("Modification times have changed, need to update cache.");
        return updated == timestamp_hash;
}

Modification of mtime of any of the directories above is sufficient to cause this to differ.

I've patched systemd to tell us why systemd thinks it needs to be reloaded (additional printfs) to get an idea what might be the trigger that is left stale.

[Zygmunt Krynicki (zyga)]

Both before and after daemon-reload the units have the same definition:

$ systemctl cat apt-news.service esm-cache.service
# /usr/lib/systemd/system/apt-news.service
# APT News is hosted at https://motd.ubuntu.com/aptnews.json and can include
# timely information related to apt updates available to your system.
# This service runs in the background during an `apt update` to download the
# latest news and set it to appear in the output of the next `apt upgrade`.
# The script won't do anything if you've run: `pro config set apt_news=false`.
# The script will limit network requests to at most once per 24 hours.
# You can also host your own aptnews.json and configure your system to use it
# with the command:
# `pro config set apt_news_url=https://yourhostname/path/to/aptnews.json`

[Unit]
Description=Update APT News

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /usr/lib/ubuntu-advantage/apt_news.py
AppArmorProfile=ubuntu_pro_apt_news
CapabilityBoundingSet=~CAP_SYS_ADMIN
CapabilityBoundingSet=~CAP_NET_ADMIN
CapabilityBoundingSet=~CAP_NET_BIND_SERVICE
CapabilityBoundingSet=~CAP_SYS_PTRACE
CapabilityBoundingSet=~CAP_NET_RAW
PrivateTmp=true
RestrictAddressFamilies=~AF_NETLINK
RestrictAddressFamilies=~AF_PACKET
# These may break some tests, and should be enabled carefully
#NoNewPrivileges=true
#PrivateDevices=true
#ProtectControlGroups=true
# ProtectHome=true seems to reliably break the GH integration test with a lunar lxd on jammy host
#ProtectHome=true
#ProtectKernelModules=true
#ProtectKernelTunables=true
#ProtectSystem=full
#RestrictSUIDSGID=true
# Unsupported in bionic
# Suggestion from systemd.exec(5) manpage on SystemCallFilter
#SystemCallFilter=@system-service
#SystemCallFilter=~@mount
#SystemCallErrorNumber=EPERM
#ProtectClock=true
#ProtectKernelLogs=true

# /usr/lib/systemd/system/esm-cache.service
# The ESM apt cache will maintain information about what ESM updates are
# available to a system. This information will be presented to users in the apt
# output, or when running pro security-status. These caches are maintained
# entirely outside the system apt configuration to avoid interference with user
# definitions. This service updates those caches. This will only have effect
# on releases where ESM is applicable, starting from Xenial: esm-apps for
# every LTS, and esm-infra for systems in expanded support period after the LTS
# expires.

[Unit]
Description=Update the local ESM caches

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /usr/lib/ubuntu-advantage/esm_cache.py

[Grant Orndorff (orndorffgrant)]

Thanks for all the investigation and discussion!

Just to close out the ubuntu-pro-client related questions:
ubuntu-pro-client does run daemon-reload in postinst.
and here is a reproducer that doesn't involve ubuntu-pro-client services

```
lxc launch ubuntu-daily:noble test
lxc shell test
# now in the noble container
cat > /usr/lib/systemd/system/hello.service << EOF
[Unit]
Description=Hello

[Service]
Type=oneshot
ExecStart=echo hello
EOF
systemctl start hello
systemctl status hello
snap install snapd
systemctl start hello # this will show the warning
systemctl cat hello.service # no noticeable change
```

So I'll mark this invalid for u-a-t.

This also demonstrates that a totally new systemd service is affected. Does snapd iterate over all systemd units to check something? Then maybe it is accidentally updating mtime even though it doesn't change contents?

[Zygmunt Krynicki (zyga)]

Yes, I think we may be enumerating a directory / statting files. I don't believe we open anything unless we want to have a look but I _could_ be wrong and I'm still investigating things (with interruptions to attend calls).

I don't believe it is related to ubuntu-pro-client, the only reason it is in the report is that "apt update" hook calls into systemctl so the warning is printed there.

[Heinrich Schuchardt (xypron)]

Today I saw the warning on an riscv64 Ubuntu 24.04 system booted from https://cdimage.ubuntu.com/ubuntu-server/daily-preinstalled/pending/noble-preinstalled-server-riscv64+icicle.img.xz after executing apt-get update.

[Eccentric Orange (eccentricorange)]

I am seeing this message on Ubuntu 24.04 Beta x64. I did an apt update and apt upgrade yesterday without any issues, and got this message out of the blue today.

Sorry, I might not have been able to follow this entire discussion, but if you need any logs/info from me and can guide me on providing them, I'll happily oblige.

**Edit:** Reboot fixed it

[Islam (islam)]

Same thing on 24.04 and rebooting doesn't fix it.

Seems those unit files belongs to: ubuntu-pro-client

[Christian Ehrhardt  (paelzer)]

TL;DR:

As many I've gone deeper, but none of the times in `stat` nor the
checksums of /usr/lib/systemd/system/hello.service did change.

Turns out this wasn't even about their file states.
And additionally my understanding was wrong, and potentially yours as well.

The state if this is outdated is not only per service via `struct UnitStatusInfo`,
but also globally across all services via `struct Manager`.

Snapd's way to enable its mount unit sets that global state and
therefore either needs to change how it enables units or run
a daemon-reload afterwards just like most .deb package installs do.

Details:

First we need to be careful, there are two ambiguous paths here that
can trigger the same message:

a) on service start
  start_unit_one
    -> if (need_daemon_reload(bus, name) > 0)
      -> warn_unit_file_changed(name);
  This is a function:
    int need_daemon_reload(sd_bus *bus, const char *unit)
  It will call out via dbus asking for the attribute NeedDaemonReload

b) on service status
  print_status_info
   -> if (i->need_daemon_reload)
      -> warn_unit_file_changed(i->id);

Also for storage, there are two:

c) `struct Manager` containing `unit_file_state_outdated` which is a global
  state for all units in that manager

d) Each `struct UnitStatusInfo` has a field `need_daemon_reload` (yes just
  named like the function above) that can flag this per unit.

And (a) isn't even per service.

The value of that can be fetched per service via dbus like:
$ dbus-send --system --print-reply --dest="org.freedesktop.systemd1" "/org/freedesktop/systemd1/unit/hello_2eservice" "org.freedesktop.DBus.Properties.Get" "string:org.freedesktop.systemd1.Unit" "string:NeedDaemonReload"
method return time=1713787033.411485 sender=:1.3 -> destination=:1.19 serial=3833 reply_serial=2
   variant boolean false

And the same can be fetched via `systemctl show` as well:

root@test:~# systemctl show hello | grep '^NeedDaemonReload'
NeedDaemonReload=yes

With the above in mind we can see that installing snapd renders ALL of them
as outdated. It was spotted with pro, reproduced with a simple example
and if you check the system it is all of them.

root@test:~# for u in $(systemctl list-units --output json | jq '.[].unit' | tr -d '"'); do systemctl show $u | grep '^NeedDaemonReload'; done 2>/dev/null | uniq -c
    143 NeedDaemonReload=no
root@test:~# snap install snapd
2024-04-22T12:44:50Z INFO Waiting for automatic snapd restart...
snapd 2.62 from Canonical✓ installed
root@test:~# for u in $(systemctl list-units --output json | jq '.[].unit' | tr -d '"'); do systemctl show $u | grep '^NeedDaemonReload'; done 2>/dev/null | uniq -c
    144 NeedDaemonReload=yes

Still the question is, which of the two data points is it switching?
It could be the global setting, but as well iterating and setting it per service.

I found that the global state could get changed in src/core/dbus-manager.c
in very similarly named methods:
- method_add_dependency_unit_files
- method_preset_all_unit_files
- method_revert_unit_files
- method_disable_unit_files_generic
- method_preset_unit_files_with_mode
- method_enable_unit_files_generic

All of them do eventually the same:
  m->un...

[J (picea-sitchensis)]

Hello. I'm also running into this problem:

sudo apt update && sudo apt upgrade
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Hit:1 http://us.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:5 https://linux.teamviewer.com/deb stable InRelease
Hit:6 https://esm.ubuntu.com/apps/ubuntu noble-apps-security InRelease
Hit:7 https://esm.ubuntu.com/infra/ubuntu noble-infra-security InRelease
Ign:8 https://repository.mullvad.net/deb/stable noble InRelease
Hit:9 https://ppa.launchpadcontent.net/unit193/encryption/ubuntu noble InRelease
Err:10 https://repository.mullvad.net/deb/stable noble Release
  404 Not Found [IP: 45.149.104.1 443]
Reading package lists... Done
W: https://ppa.launchpadcontent.net/unit193/encryption/ubuntu/dists/noble/InRelease: Signature by key 3BFB8E06536B8753AC58A4A303647209B58A653A uses weak algorithm (rsa1024)
E: The repository 'https://repository.mullvad.net/deb/stable noble Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

I have no idea what it even means. Any suggestions? Should I even be concerned about this?

[Bass Ackwards (bassackwards)]

Fixed
Fixed
Fixed
Fixed after reboot.
Did reboot work for any others?

Happened to me after clearing out games,then apt purge hexchat.

Machine is all enterprise gear. Dell poweredge r730xd
my cli error is:

sudo apt update
[sudo] password for xxxxxx:
Sorry, try again.
[sudo] password for xxxxxx:
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Hit:1 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:2 http://ca.archive.ubuntu.com/ubuntu noble InRelease
Get:3 http://ca.archive.ubuntu.com/ubuntu noble-updates InRelease [89.7 kB]
Hit:4 http://ca.archive.ubuntu.com/ubuntu noble-backports InRelease
Fetched 89.7 kB in 1s (113 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

[Grant Orndorff (orndorffgrant)]

It looks like the global need-reload state that Christian investigated that is being set by a snapd operation was added recently in systemd.

https://github.com/systemd/systemd/commit/a82b8b3dc80619c3275ad8180069289b411206d0

That is likely why we're only seeing this issue in noble.

From reading the commit message there, it sounds like the right thing to do is for snapd to issue a daemon-reload after it sets up all its units.

[Paul White (paulw2u)]

I'm sorry if this comment isn't helpful but I'm only seeing this on a new noble installation but never on another installation which was upgraded from mantic some time ago but early on during the noble development period.

[corrado venturini (corradoventu)]

Also in Ubuntu 24.10 Oracular installed from ISO, not upgraded from Noble:
corrado@corrado-n8-oo-0509:~$ sudo apt update && sudo apt upgrade
[sudo] password for corrado:
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Hit:1 http://security.ubuntu.com/ubuntu oracular-security InRelease
Get:2 http://archive.ubuntu.com/ubuntu oracular InRelease [73.9 kB]
.....

[Stephen Waines (stevinski)]

Thank you but I never reported this bug

On May 20, 2024 9:40:38 a.m. corrado venturini <email address hidden>
wrote:

> Also in Ubuntu 24.10 Oracular installed from ISO, not upgraded from Noble:
> corrado@corrado-n8-oo-0509:~$ sudo apt update && sudo apt upgrade
> [sudo] password for corrado:
> Warning: The unit file, source configuration file or drop-ins of
> apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload
> units.
> Warning: The unit file, source configuration file or drop-ins of
> esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload
> units.
> Hit:1 http://security.ubuntu.com/ubuntu oracular-security InRelease
> Get:2 http://archive.ubuntu.com/ubuntu oracular InRelease [73.9 kB]
> .....
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (2065717).
> https://bugs.launchpad.net/bugs/2055239
>
> Title:
> Warning: The unit file, source configuration file or drop-ins of {apt-
> news,esm-cache}.service changed on disk. Run 'systemctl daemon-reload'
> to reload units.
>
> Status in snapd:
> New
> Status in ubuntu-advantage-tools package in Ubuntu:
> Invalid
>
> Bug description:
> I recently started seeing the following warning messages when I run
> `apt update`.
>
> $ sudo apt update
> Warning: The unit file, source configuration file or drop-ins of
> apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload
> units.
> Warning: The unit file, source configuration file or drop-ins of
> esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload
> units.
> ...
>
> apt-news.service for example is in /lib/systemd/system/apt-
> news.service and it's a static file managed by the package. Does the
> package maintenance script call systemd related hooks to reload the
> config whenever the package gets updated?
>
> $ systemctl cat apt-news.service
> # /usr/lib/systemd/system/apt-news.service
> # APT News is hosted at https://motd.ubuntu.com/aptnews.json and can include
> # timely information related to apt updates available to your system.
> ...
>
> $ dpkg -S /lib/systemd/system/apt-news.service
> ubuntu-pro-client: /lib/systemd/system/apt-news.service
>
> ProblemType: BugDistroRelease: Ubuntu 24.04
> Package: ubuntu-pro-client 31.1
> ProcVersionSignature: Ubuntu 6.6.0-14.14-generic 6.6.3
> Uname: Linux 6.6.0-14-generic x86_64
> NonfreeKernelModules: zfs
> ApportVersion: 2.28.0-0ubuntu1
> Architecture: amd64
> CasperMD5CheckResult: pass
> CurrentDesktop: ubuntu:GNOME
> Date: Wed Feb 28 13:06:35 2024
> InstallationDate: Installed on 2024-01-08 (51 days ago)
> InstallationMedia: Ubuntu 24.04 LTS "Noble Numbat" - Daily amd64 (20240104)
> ProcEnviron:
> LANG=en_US.UTF-8
> PATH=(custom, no user)
> SHELL=/bin/bash
> TERM=xterm-256color
> XDG_RUNTIME_DIR=<set>SourcePackage: ubuntu-advantage-tools
> UpgradeStatus: No upgrade log present (probably fresh install)
> apparmor_logs.txt:
>
> cloud-id.txt-error:
> Failed running command 'cloud-id' [exit(2)]. Message: REDACTED config part
> /etc/cloud/cloud.cfg.d/99-installer.cfg, insufficient permissions
> REDACTED config...

[Phil O (po5857)]

Possibly related to this:

https://github.com/systemd/systemd/issues/32959

[Derek (derek-shnosh)]

This just started showing up for me today. I upgraded from 23.10 > 24.04 last week.

[Grant Orndorff (orndorffgrant)]

That systemd issue does appear to be the root cause

[Nick Rosbrook (enr0n)]

This issue can be seen with any actual unit after e.g. enabling or disabling a non-existent unit. For example,

root@n:~# systemctl status ssh.socket
● ssh.socket - OpenBSD Secure Shell server socket
     Loaded: loaded (/usr/lib/systemd/system/ssh.socket; enabled; preset: enabled)
     Active: active (listening) since Wed 2024-07-31 19:51:18 UTC; 5min ago
   Triggers: ● ssh.service
     Listen: [::]:22 (Stream)
     CGroup: /system.slice/ssh.socket

Jul 31 19:55:15 n systemd[1]: ssh.socket: Trying to enqueue job ssh.socket/start/replace
Jul 31 19:55:15 n systemd[1]: ssh.socket: Installed new job ssh.socket/start as 734
Jul 31 19:55:15 n systemd[1]: ssh.socket: Enqueued job ssh.socket/start as 734
Jul 31 19:55:15 n systemd[1]: ssh.socket: Job 734 ssh.socket/start finished, result=done
Jul 31 19:55:53 n systemd[1]: ssh.socket: Changed dead -> listening
Jul 31 19:56:05 n systemd[1]: ssh.socket: Trying to enqueue job ssh.socket/start/replace
Jul 31 19:56:05 n systemd[1]: ssh.socket: Installed new job ssh.socket/start as 825
Jul 31 19:56:05 n systemd[1]: ssh.socket: Enqueued job ssh.socket/start as 825
Jul 31 19:56:05 n systemd[1]: ssh.socket: Job 825 ssh.socket/start finished, result=done
Jul 31 19:56:20 n systemd[1]: ssh.socket: Changed dead -> listening

root@n:~# systemctl enable fdfldakfjdklfjdaljfa.service
Failed to enable unit: Unit file fdfldakfjdklfjdaljfa.service does not exist.

root@n:~# systemctl status ssh.socket
Warning: The unit file, source configuration file or drop-ins of ssh.socket changed on disk. Run 'systemctl daemon-reload' to reload units.
● ssh.socket - OpenBSD Secure Shell server socket
     Loaded: loaded (/usr/lib/systemd/system/ssh.socket; enabled; preset: enabled)
     Active: active (listening) since Wed 2024-07-31 19:51:18 UTC; 5min ago
   Triggers: ● ssh.service
     Listen: [::]:22 (Stream)
     CGroup: /system.slice/ssh.socket

Jul 31 19:55:15 n systemd[1]: ssh.socket: Trying to enqueue job ssh.socket/start/replace
Jul 31 19:55:15 n systemd[1]: ssh.socket: Installed new job ssh.socket/start as 734
Jul 31 19:55:15 n systemd[1]: ssh.socket: Enqueued job ssh.socket/start as 734
Jul 31 19:55:15 n systemd[1]: ssh.socket: Job 734 ssh.socket/start finished, result=done
Jul 31 19:55:53 n systemd[1]: ssh.socket: Changed dead -> listening
Jul 31 19:56:05 n systemd[1]: ssh.socket: Trying to enqueue job ssh.socket/start/replace
Jul 31 19:56:05 n systemd[1]: ssh.socket: Installed new job ssh.socket/start as 825
Jul 31 19:56:05 n systemd[1]: ssh.socket: Enqueued job ssh.socket/start as 825
Jul 31 19:56:05 n systemd[1]: ssh.socket: Job 825 ssh.socket/start finished, result=done
Jul 31 19:56:20 n systemd[1]: ssh.socket: Changed dead -> listening

I have tested [1] in a PPA build, and the issue goes away.

It may have already been pointed out, but the reason this is seen with apt-news.service and esm-cache.service specifically is because of the hook defined in /etc/apt/apt.conf.d/20apt-esm-hook.conf which starts these services. Besides that, there is nothing special about them which makes this bug appear.

[1] https://github.com/systemd/systemd/pull/32610/commits/d69cba3bfffc7b1e3197e2a34b459db13b1e1cb7

[Nick Rosbrook (enr0n)]

Note that the patch is already present in v256, so this only needs to be fixed in Noble.

[phm2f1alcw69 (phm2f1alcw69)]

root@xxx:/root# aptitude update
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[phm2f1alcw69 (phm2f1alcw69)]

Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
Linux tox250 6.8.0-39-generic #39-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 5 21:49:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[Brian Murray (brian-murray)]

Hello Nobuto, or anyone else affected,

Accepted systemd into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/255.4-1ubuntu8.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brett Nelson (blizzardville)]

Put a fresh Noble on a few days ago and so far had only installed one program besides what you get during standard during the standard install (offline app option + non-free boxes checked).

Ran into the problem being described here with the warning messages during sudo apt update.

Per Brian Murray's suggestion I enabled -proposed and rebooted (had tried rebooting before to no effect).

Problem appears to be solved, no more warnings.

One additional note, when I went through the OS install I enabled Ubuntu Pro with my Ubuntu One account but didn't get any feedback like it hadn't finished doing it so opened a terminal and ran the Ubuntu Pro enable from the command line. It said already had been enabled. Sorry if wording is not precise but just posting this feedback. Thanks for fixing these things, Ubuntu has been my favorite OS for more than a decade and really appreciate those who work on it.

Brett

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (255.4-1ubuntu8.3) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

389-ds-base/unknown (s390x)
aide/unknown (s390x)
apport/2.28.1-0ubuntu3 (arm64, s390x)
appstream/unknown (s390x)
apt/unknown (armhf, s390x)
asterisk/unknown (armhf)
at-spi2-core/unknown (armhf)
bind9/unknown (s390x)
bolt/unknown (s390x)
casper/unknown (armhf)
casync/unknown (s390x)
ceph/unknown (s390x)
certspotter/unknown (s390x)
clevis/unknown (armhf, s390x)
clutter-1.0/1.26.4+git2779b932+dfsg-7build3 (armhf)
collectd/unknown (s390x)
comitup/unknown (s390x)
corosync/unknown (armhf)
corosync-qdevice/unknown (s390x)
coturn/4.6.1-1build4 (armhf)
coturn/unknown (s390x)
cron/3.0pl1-184ubuntu2 (ppc64el)
csync2/unknown (s390x)
cups/unknown (s390x)
dbus/1.14.10-4ubuntu4 (arm64)
debci/unknown (armhf)
dhcpcd/unknown (s390x)
dovecot/1:2.3.21+dfsg1-2ubuntu5 (arm64)
exim4/4.97-4ubuntu4.1 (ppc64el)
fetchmail/6.4.38-1ubuntu4 (arm64)
fetchmail/unknown (armhf)
fluidsynth/unknown (armhf, s390x)
freedom-maker/0.33 (armhf)
freedom-maker/unknown (arm64)
gvfs/1.54.0-1ubuntu2 (s390x)
haproxy/unknown (armhf)
incus/unknown (armhf)
init-system-helpers/unknown (s390x)
initramfs-tools/unknown (armhf)
libcamera/unknown (armhf, s390x)
libinput/unknown (s390x)
libreswan/unknown (arm64, s390x)
libsdl2/unknown (s390x)
libsfml/unknown (s390x)
libusbauth-configparser/unknown (armhf)
libvirt/unknown (armhf)
lighttpd/unknown (s390x)
mariadb/1:10.11.8-0ubuntu0.24.04.1 (arm64, armhf, s390x)
mediawiki/1:1.39.7-1 (amd64, armhf)
monitoring-plugins-systemd/unknown (armhf)
mosquitto/2.0.18-1build3 (arm64, armhf)
mpd/0.23.14-2build5 (armhf)
munin/unknown (s390x)
mutter/unknown (arm64)
ndctl/unknown (s390x)
netplan.io/1.0-2ubuntu1.2 (arm64, s390x)
nftables/unknown (s390x)
nghttp2/unknown (armhf)
nix/unknown (armhf)
nut/unknown (s390x)
openbsd-inetd/unknown (s390x)
openssh/1:9.6p1-3ubuntu13.4 (s390x)
ostree/2024.5-1build2 (s390x)
pam/unknown (s390x)
pdns-recursor/unknown (s390x)
php8.3/8.3.6-0ubuntu0.24.04.1 (armhf)
php8.3/unknown (s390x)
policykit-1/124-2ubuntu1 (amd64)
polkit-qt-1/0.200.0-1build4 (armhf)
polkit-qt-1/unknown (s390x)
postgresql-16/16.3-0ubuntu0.24.04.1 (armhf)
procps/2:4.0.4-4ubuntu3 (armhf)
prometheus-homeplug-exporter/unknown (s390x)
prometheus-ipmi-exporter/1.8.0-1ubuntu0.24.04.1 (armhf)
prometheus-ipmi-exporter/unknown (s390x)
prometheus-postfix-exporter/unknown (s390x)
prometheus-squid-exporter/1.10.4+ds-1ubuntu0.24.04.1 (armhf)
pulseaudio/unknown (armhf, s390x)
puppet-agent/8.4.0-1 (armhf)
remctl/3.18-1.1build2 (armhf)
rsyslog/8.2312.0-3ubuntu9 (arm64)
rust-whoami/unknown (s390x)
samba/2:4.19.5+dfsg-4ubuntu9 (s390x)
sbws/unknown (armhf)
seatd/unknown (armhf, s390x)
snapd/unknown (s390x)
swupdate/2023.12.1+dfsg-1ubuntu5 (armhf)
swupdate/unknown (s390x)
systemd-cron/unknown (armhf)
systemd-hwe/unknown (armhf)
upower/1.90.3-1 (arm64)
upower/unknown (armhf)
usbauth/unknown (armhf)
util-linux/unknown (arm64)
webhook/unknown (s390x)
wireshark/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy r...

[Nick Rosbrook (enr0n)]

I have verified the fix using systemd 255.4-1ubuntu3.3 from noble-proposed.

In a new container, I can see the issue at first:

nr@six:~$ lxc launch ubuntu:noble lp2055239
Launching lp2055239
nr@six:~$ lxc exec lp2055239 bash
root@lp2055239:~# systemctl enable somethingthatdefinitelydoesnotexist.service
Failed to enable unit: Unit file somethingthatdefinitelydoesnotexist.service does not exist.
root@lp2055239:~# systemctl status systemd-resolved.service
Warning: The unit file, source configuration file or drop-ins of systemd-resolved.service changed on disk. Run>
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-08-08 13:34:53 UTC; 43s ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 176 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 18947)
     Memory: 2.6M (peak: 3.3M)
        CPU: 168ms
     CGroup: /system.slice/systemd-resolved.service
             └─176 /usr/lib/systemd/systemd-resolved

Aug 08 13:34:52 lp2055239 systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Aug 08 13:34:52 lp2055239 systemd-resolved[176]: Positive Trust Anchors:
Aug 08 13:34:52 lp2055239 systemd-resolved[176]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bb>
Aug 08 13:34:52 lp2055239 systemd-resolved[176]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-a>
Aug 08 13:34:53 lp2055239 systemd-resolved[176]: Using system hostname 'lp2055239'.
Aug 08 13:34:53 lp2055239 systemd[1]: Started systemd-resolved.service - Network Name Resolution.
root@lp2055239:~# apt update
Warning: The unit file, source configuration file or drop-ins of apt-news.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: The unit file, source configuration file or drop-ins of esm-cache.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Get:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:3 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [15.0 MB]
0% [5 Packages 8229 kB/15.0 MB 55%]^C

Then, I enabled noble-proposed, upgraded systemd, and the issue was gone:

root@lp2055239:~# cat > /etc/apt/sources.list.d/noble-proposed.sources << EOF
> Types: deb
URIs: http://us.archive.ubuntu.com/ubuntu/
Suites: noble-proposed
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
> EOF
root@lp2055239:~# apt update

... SNIP ...

root@lp2055239:~# apt policy systemd
systemd:
  Installed: 255.4-1ubuntu8.2
  Candidate: 255.4-1ubuntu8.2
  Version table:
     255.4-1ubuntu8.3 100
       ...

[Łukasz Zemczak (sil2100)]

Hello Nobuto, or anyone else affected,

Accepted systemd into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/255.4-1ubuntu8.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

The new upload is just a no-change rebuild for new build flags, so my previous verification stands.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (255.4-1ubuntu8.4) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

aide/unknown (armhf)
apport/unknown (armhf)
apt/2.7.14build2 (s390x)
casync/unknown (armhf)
cockpit/unknown (armhf)
comitup/1.39-1 (armhf)
dhcpcd/unknown (armhf)
dropbear/unknown (armhf)
freeradius/unknown (armhf)
golang-github-coreos-go-systemd/unknown (armhf)
initramfs-tools/0.142ubuntu25.1 (armhf)
libcamera/0.2.0-3fakesync1build6 (armhf)
libsoup3/3.4.4-5build2 (armhf)
libvirt/10.0.0-2ubuntu8.3 (armhf)
mariadb/unknown (armhf)
netplan.io/1.0-2ubuntu1.2 (arm64, armhf)
nghttp2/unknown (armhf)
nix/unknown (armhf)
ovn/24.03.2-0ubuntu0.24.04.1 (arm64)
pgagroal/unknown (armhf)
policykit-1/unknown (armhf)
remctl/unknown (armhf)
sks/1.1.6+git20210302.c3ba6d5a-4build2 (armhf)
snapd/2.63+24.04ubuntu0.1 (s390x)
speech-dispatcher/unknown (armhf)
systemd-hwe/unknown (armhf)
tang/unknown (armhf)
tmux/unknown (armhf)
util-linux/2.39.3-9ubuntu6 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[corrado venturini (corradoventu)]

Still same problem on Ubuntu 24.10 Oracular.

[Nick Rosbrook (enr0n)]

@corradoventu, can you provide some evidence of that? I cannot reproduce this on 24.10.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 255.4-1ubuntu8.4

---------------
systemd (255.4-1ubuntu8.4) noble; urgency=medium

  * No-change rebuild to pick up changed build flags on ppc64el and s390x.
    (LP: #2076340)

systemd (255.4-1ubuntu8.3) noble; urgency=medium

  * core/dbus-manager: mark unit file state as outdated only if some changes succeeded
    (LP: #2055239)

 -- Nick Rosbrook <email address hidden> Thu, 08 Aug 2024 10:51:10 -0400

[Sunney Xue (sunneyxue)]

Still have this issue on ubuntu 24.04, with systemd 255 (255.4-1ubuntu8.4).

Strange enough, this warning message doesn't show up every time I run "sudo apt update". It appears rather randomly. And as it indicted "systemctl daemon-reload" will get rid of this message, at least for a while.

System info:

# System Details Report
---

## Report details
- **Date generated:** 2024-08-23 14:47:19

## Hardware Information:
- **Hardware Model:** Apple Inc. MacBookAir7,1
- **Memory:** 8.0 GiB
- **Processor:** Intel® Core™ i7-5650U × 4
- **Graphics:** Intel® HD Graphics 6000 (BDW GT3)
- **Disk Capacity:** 500.3 GB

## Software Information:
- **Firmware Version:** 481.0.0.0.0
- **OS Name:** Ubuntu 24.04 LTS
- **OS Build:** (null)
- **OS Type:** 64-bit
- **GNOME Version:** 46
- **Windowing System:** Wayland
- **Kernel Version:** Linux 6.8.0-41-generic

[Nick Rosbrook (enr0n)]

@sunneyxue, can you share the output of the following?

$ apt policy systemd
$ sudo systemctl daemon-reload
$ sudo systemctl enable somethingthatdefinitelydoesnotexist.service
$ sudo apt update

[corrado venturini (corradoventu)]

Still same problem in Ubuntu 24.10
corrado@corrado-n13-oo-0823:~$ apt policy systemd
systemd:
  Installed: 256.4-2ubuntu1
  Candidate: 256.4-2ubuntu1
  Version table:
 *** 256.4-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu oracular/main amd64 Packages
        100 /var/lib/dpkg/status
corrado@corrado-n13-oo-0823:~$ sudo systemctl enable somethingthatdefinitelydoesnotexist.service
Failed to enable unit: Unit somethingthatdefinitelydoesnotexist.service does not exist
corrado@corrado-n13-oo-0823:~$ sudo apt update
Hit:1 http://security.ubuntu.com/ubuntu oracular-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu oracular InRelease
Hit:3 http://archive.ubuntu.com/ubuntu oracular-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu oracular-backports InRelease
All packages are up to date.
corrado@corrado-n13-oo-0823:~$

[Sunney Xue (sunneyxue)]

@Nick. Here is the output:

xxx@xxx:~$ apt policy systemd
systemd:
  Installed: 255.4-1ubuntu8.4
  Candidate: 255.4-1ubuntu8.4
  Version table:
 *** 255.4-1ubuntu8.4 500
        500 http://nl.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     255.4-1ubuntu8 500
        500 http://nl.archive.ubuntu.com/ubuntu noble/main amd64 Packages
xxx@xxx:~$ sudo systemctl daemon-reload
xxx@xxx:~$ sudo systemctl enable somethingthatdefinitelydoesnotexist.service
Failed to enable unit: Unit file somethingthatdefinitelydoesnotexist.service does not exist.
xxx@xxx:~$ sudo apt update
Hit:1 http://nl.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:3 http://nl.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:4 http://nl.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:5 https://packages.microsoft.com/repos/code stable InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.
xxx@xxx:~$ sudo apt list --upgradable
Listing... Done
gnome-text-editor/noble-updates 46.3-0ubuntu1 amd64 [upgradable from: 46.1-1]
language-pack-en-base/noble-updates 1:24.04+20240817 all [upgradable from: 1:24.04+20240419]
language-pack-en/noble-updates 1:24.04+20240817 all [upgradable from: 1:24.04+20240419]
language-pack-gnome-en-base/noble-updates 1:24.04+20240817 all [upgradable from: 1:24.04+20240419]
language-pack-gnome-en/noble-updates 1:24.04+20240817 all [upgradable from: 1:24.04+20240419]
libssl3t64/noble-updates 3.0.13-0ubuntu3.3 amd64 [upgradable from: 3.0.13-0ubuntu3.2]
openssl/noble-updates 3.0.13-0ubuntu3.3 amd64 [upgradable from: 3.0.13-0ubuntu3.2]

[Nick Rosbrook (enr0n)]

@corradoventu - the output you shared demonstrates that you are *not* seeing this issue on 24.10.

@sunneyxue- likewise, your output demonstrates that the issue is fixed on this system.

Note that it is possible at the time when you run `apt update`, that a daemon-reload actually is needed, and the warning will *correctly* be displayed. If we want this to always be hidden when running apt update, the apt hook needs to be updated to pass --no-warn to the systemctl call.

[Grant Orndorff (orndorffgrant)]

Thanks @enr0n. We had already created a separate bug (bug 2074211) to represent the case that you describe, so I'll mark this one back to "Invalid" for ubuntu-advantage-tools.

[Sunney Xue (sunneyxue)]

@Nick, as I mentioned in my first post "This warning message doesn't show up every time I run "sudo apt update". It appears rather randomly. And as it indicted "systemctl daemon-reload" will get rid of this message, at least for a while."

I haven't seen this warning in the past two days.