debsums reports file changed /usr/lib/systemd/system/cloud-init.service

[Cyrus Lien]

[Summary]
debsums reported /lib/systemd/system/cloud-init.service was changed.
Also the cloud-init.service mentioned it was auto-generated by installer build process.

The diff output from package included and system installed.
$ diff ./extract-cloud-init/lib/systemd/system/cloud-init.service /lib/systemd/system/cloud-init.service
0a1,2
> # Automatically generated by installer build process
> # Based on cloud-init 23.1 for Desktop LiveCD
8a11,13
> # Installer Added After=NetworkManager* ordering
> After=NetworkManager.service
> After=NetworkManager-wait-online.service
13c18
< Before=sysinit.target
---
> # Before=sysinit.target Installer removed to allow for NM dependency
16,18c21
< ConditionPathExists=!/etc/cloud/cloud-init.disabled
< ConditionKernelCommandLine=!cloud-init=disabled
< ConditionEnvironment=!KERNEL_CMDLINE=cloud-init=disabled
---
> Before=systemd-user-sessions.service

[Steps to reproduce]
1. Install Ubuntu 24.04 desktop
2. Excute command `debsums -c`

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: cloud-init 24.1.3-0ubuntu3
ProcVersionSignature: Ubuntu 6.8.0-35.35-generic 6.8.4
Uname: Linux 6.8.0-35-generic x86_64
ApportVersion: 2.28.1-0ubuntu2
Architecture: amd64
AutoInstallUserData: Error: [Errno 13] Permission denied: '/var/log/installer/autoinstall-user-data'
CasperMD5CheckResult: pass
CurtinConfigCurtHooks: Error: [Errno 13] Permission denied: '/var/log/installer/curtin-install/subiquity-curthooks.conf'
CurtinConfigExtract: Error: [Errno 13] Permission denied: '/var/log/installer/curtin-install/subiquity-extract.conf'
CurtinConfigInitial: Error: [Errno 13] Permission denied: '/var/log/installer/curtin-install/subiquity-initial.conf'
CurtinConfigPartitioning: Error: [Errno 13] Permission denied: '/var/log/installer/curtin-install/subiquity-partitioning.conf'
Date: Fri Jun 14 13:57:57 2024
InstallationDate: Installed on 2024-05-02 (43 days ago)
InstallationMedia: Ubuntu 24.04 LTS "Noble Numbat" - Release amd64 (20240424)
InstallerCloudCfg: Error: [Errno 13] Permission denied: '/etc/cloud/cloud.cfg.d/99-installer.cfg'
PackageArchitecture: all
SourcePackage: cloud-init
SubiquityServerDebug: Error: [Errno 13] Permission denied: '/var/log/installer/subiquity-server-debug.log.3103'
UpgradeStatus: No upgrade log present (probably fresh install)
cloud-init-log-warnings:
 2024-05-02 07:52:44,723 - activators.py[WARNING]: Running ['netplan', 'apply'] resulted in stderr output:
 ** (generate:1042): WARNING **: 07:52:44.418: Permissions for /etc/netplan/01-network-manager-all.yaml are too open. Netplan configuration should NOT be accessible by others.
 ** (process:1041): WARNING **: 07:52:44.646: Permissions for /etc/netplan/01-network-manager-all.yaml are too open. Netplan configuration should NOT be accessible by others.
 ** (process:1041): WARNING **: 07:52:44.696: Permissions for /etc/netplan/01-network-manager-all.yaml are too open. Netplan configuration should NOT be accessible by others.
 2024-05-02 07:53:41,599 - cc_final_message.py[WARNING]: Used fallback datasource
user_data.txt: Error: [Errno 13] Permission denied: '/var/lib/cloud/instances/iid-datasource-none/user-data.txt'

[Brett Holman (holmanb)]

Thanks for reporting this issue! By the changed file it looks like something (subiquity?) is changing one of cloud-init's service files.

Since it looks like this is an issue in an external project, I'm setting this issue to Invalid on the cloud-init project for now.

[Yao Wei (medicalwei)]

This seems to be the issue caused by livecd-rootfs:

livecd-rootfs (2.816) lunar; urgency=medium

  [ Chad Smith ]
  * Replace cloud-init.service on NetworkManager images to address an ordering
    cycle interfering with autoinstall with ubuntu-desktop-installer.
    (LP: #2008952)

 -- Dan Bungert <email address hidden> Fri, 24 Mar 2023 18:35:46 -0600

Is there something to be fixed on either side?

[Brett Holman (holmanb)]

> [ Chad Smith ]
> * Replace cloud-init.service on NetworkManager images to address an ordering
> cycle interfering with autoinstall with ubuntu-desktop-installer.
> (LP: #2008952)

It sounds like you've correctly diagnosed the root cause.

> Is there something to be fixed on either side?

The change was to fix an issue in the installer - but the fix rewrites the .service file provided by a different package. I'm not sure if debian packaging handles this in a safe (ordered) way, but assuming it doesn't (as evidenced by the debsums error) then yes - this should probably be fixed.

To fix this packaging issue, livecd-rootfs could instead provide a drop-in config in, for example, /lib/systemd/system/cloud-init.service.d/10-override.conf, which modifies the cloud-init.service's boot order.

[Dan Bungert (dbungert)]

Marking invalid for Subiquity as I expect we're addressing this in livecd-rootfs.
An override file sounds like a good direction if it's flexible enough for this case.

[Chad Smith (chad.smith)]

@Dan Bungert. Since this was my changeset, I thought I'd put up a PR as food for thought which uses the override that Brett mentioned. It avoids the wholesale replacement of the entire unit file and just addresses Before and After config in the supplemental override file.

I also updated the After snapshot to align with cloud-init 24.2 which added a Before=systemd-user-sessions.service

https://code.launchpad.net/~chad.smith/livecd-rootfs/+git/livecd-rootfs/+merge/469479

Note that cloud-init SRU of 24.2 is queued in unapproved queue for focal/jammy and noble
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2071762

[Alberto Contreras (aciba)]

We discussed in cloud-init stand-up, about the following potential issue:

Already installed servers / desktops with the current livecd-rootfs, i.e. containing the modified /usr/lib/systemd/system/cloud-init.service, could get conflicts while upgrading to the latest cloud-init version.

Is this an issue? If so, do we want to try to automatically fix it for users, in some maintainer script of some package?

[Chad Smith (chad.smith)]

Thanks @alberto. I confirmed the following:

The question of whether apt upgrade would show us the Files changed prompt which would break unattended upgrades. The answer is we don't have to worry about it because apt upgrade only reacts to changes in files registered by the deb package as conffiles. Any other non-conffiles are not monitored for that change dialog. From dpkg --status cloud-init or directly at /var/lib/dpkg/info/cloud-init.conffiles we can see that our registered conffiles all live under /etc, so the fact that livecd-rootfs has a modified /lib/systemd/system/cloud-init.service conf doesn't register for apt upgrade type config changed dialogs. That said, any modified path installed which has an md5sum listed in /var/lib/dpkg/info/cloud-init.md5sums would be noted by debsums -c if the md5sum changes on those files.

So, there is nothing in cloud-init upstream packaging to do here related to this fix because a modified /lib/systemd/system/cloud-init.service doesn't trigger that prompt on apt upgrade or unattended upgrades.

[Chad Smith (chad.smith)]

1. Confirmed diff from packaged cloud-init.service and override
root@lcd-o:~# diff -urN /lib/systemd/system/cloud-init.service /etc/systemd/system/cloud-init.service
--- /lib/systemd/system/cloud-init.service 2024-06-26 18:30:42.000000000 +0000
+++ /etc/systemd/system/cloud-init.service 2024-07-18 21:37:31.201082523 +0000
@@ -1,3 +1,6 @@
+# Based on cloud-init 24.2 for Desktop LiveCD
+# Redact sysinit.target from Before, add After=NetworkManager*.service
+# (LP: #2008952)
 [Unit]
 # https://cloudinit.readthedocs.io/en/latest/explanation/boot.html
 Description=Cloud-init: Network Stage
@@ -8,11 +11,12 @@
 After=cloud-init-local.service
 After=systemd-networkd-wait-online.service
 After=networking.service
+After=NetworkManager.service NetworkManager-wait-online.service
 Before=network-online.target
 Before=sshd-keygen.service
 Before=sshd.service
 Before=systemd-user-sessions.service
-Before=sysinit.target
+# Before=sysinit.target Redacted due to NetworkManager.service dbus depends
 Before=shutdown.target
 Conflicts=shutdown.target
 ConditionPathExists=!/etc/cloud/cloud-init.disabled

2. Confirmed override shows up properly via systemctl show same functional cloud-init.service deltas
 WantedBy=cloud-init.target cloud-config.target
 Conflicts=shutdown.target
-Before=network-online.target sysinit.target cloud-config.target sshd-keygen.service systemd-user-sessions.service shutdown.target sshd.service
-After=cloud-init-local.service systemd-networkd-wait-online.service systemd-journald.socket system.slice networking.service
+Before=sshd.service shutdown.target cloud-config.target systemd-user-sessions.service network-online.target sshd-keygen.service
+After=NetworkManager-wait-online.service cloud-init-local.service system.slice systemd-journald.socket networking.service systemd-networkd-wait-online.service NetworkManager.service
 Description=Cloud-init: Network Stage
 LoadState=loaded
 ActiveState=active
 FreezerState=running
 SubState=exited

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 24.10.6

---------------
livecd-rootfs (24.10.6) oracular; urgency=medium

  [ Chad Smith ]
  * Override cloud-init.service in /etc so as to not invalidate debsums
    (LP: #2069391)
  * Update the overridden copy of cloud-init.service

 -- Dan Bungert <email address hidden> Thu, 18 Jul 2024 15:58:53 -0600