cloud-init should delete hook file in /etc/NetworkManager/dispatcher.d during upgrade

[Reinhard Munz]

During upgrade of cloud-init there are dpkg warnings that '/etc/NetworkManager' may not be deleted. The delete fails due to a hook script file of cloud-init inside a sub-folder, to be exact at '/etc/NetworkManager/dispatcher.d/hook-network-manager'.

Package upgrade should handle deletion of the hook script and thus not fail to delete the directory.

The following is the output from my command line of the upgrade and the following investigation as to which files prevented deletion of the directory and caused the warning.

rm@g0-srv-vie:~$ sudo apt -o APT::Get::Always-Include-Phased-Updates=true upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
Get more security updates through Ubuntu Pro with 'esm-apps' enabled:
  exo-utils libexo-2-0 libexo-common
Learn more about Ubuntu Pro at https://ubuntu.com/pro
#
# An OpenSSL vulnerability has recently been fixed with USN-6188-1 & 6119-1:
# CVE-2023-2650: possible DoS translating ASN.1 object identifiers.
# Ensure you have updated the package to its latest version.
#
The following packages will be upgraded:
  cloud-init
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 533 kB of archives.
After this operation, 26.6 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://at.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cloud-init all 23.2.1-0ubuntu0~22.04.1 [533 kB]
Fetched 533 kB in 0s (3,138 kB/s)
Preconfiguring packages ...
(Reading database ... 203794 files and directories currently installed.)
Preparing to unpack .../cloud-init_23.2.1-0ubuntu0~22.04.1_all.deb ...
Unpacking cloud-init (23.2.1-0ubuntu0~22.04.1) over (23.1.2-0ubuntu0~22.04.1) ...
dpkg: warning: unable to delete old directory '/etc/NetworkManager/dispatcher.d': Directory not empty
dpkg: warning: unable to delete old directory '/etc/NetworkManager': Directory not empty
Setting up cloud-init (23.2.1-0ubuntu0~22.04.1) ...
Installing new version of config file /etc/cloud/cloud.cfg ...
Installing new version of config file /etc/cloud/templates/chrony.conf.alpine.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.centos.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.cos.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.debian.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.fedora.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.freebsd.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.opensuse-leap.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.opensuse-microos.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.opensuse-tumbleweed.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.opensuse.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.photon.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.rhel.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.sle-micro.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.sle_hpc.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.sles.tmpl ...
Installing new version of config file /etc/cloud/templates/chrony.conf.ubuntu.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.alpine.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.debian.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.fedora.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.freebsd.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.opensuse.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.photon.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.rhel.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.sles.tmpl ...
Installing new version of config file /etc/cloud/templates/ntp.conf.ubuntu.tmpl ...
Installing new version of config file /etc/cloud/templates/ntpd.conf.openbsd.tmpl ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
rm@g0-srv-vie:~$ sudo ls -lah /etc/NetworkManager
total 20K
drwxr-xr-x 3 root root 4.0K Aug 9 2022 .
drwxr-xr-x 146 root root 12K Jul 15 19:24 ..
drwxr-xr-x 2 root root 4.0K Apr 28 06:44 dispatcher.d
rm@g0-srv-vie:~$ sudo ls -lah /etc/NetworkManager/dispatcher.d
total 12K
drwxr-xr-x 2 root root 4.0K Apr 28 06:44 .
drwxr-xr-x 3 root root 4.0K Aug 9 2022 ..
-rwxr-xr-x 1 root root 719 May 18 2022 hook-network-manager
rm@g0-srv-vie:~$ sudo cat /etc/NetworkManager/dispatcher.d/hook-network-manager
#!/bin/sh
# This file is part of cloud-init. See LICENSE file for license information.

# This script hooks into NetworkManager(8) via its scripts
# arguments are 'interface-name' and 'action'
#
is_azure() {
    local dmi_path="/sys/class/dmi/id/board_vendor" vendor=""
    if [ -e "$dmi_path" ] && read vendor < "$dmi_path"; then
        [ "$vendor" = "Microsoft Corporation" ] && return 0
    fi
    return 1
}

is_enabled() {
    # only execute hooks if cloud-init is enabled and on azure
    [ -e /run/cloud-init/enabled ] || return 1
    is_azure
}

if is_enabled; then
    case "$1:$2" in
        *:up) exec cloud-init dhclient-hook up "$1";;
        *:down) exec cloud-init dhclient-hook down "$1";;
    esac
fi

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: cloud-init 23.2.1-0ubuntu0~22.04.1
ProcVersionSignature: Ubuntu 5.15.0-76.83-generic 5.15.99
Uname: Linux 5.15.0-76-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: pass
Date: Sat Jul 15 19:32:09 2023
InstallationDate: Installed on 2022-11-24 (233 days ago)
InstallationMedia: Ubuntu-Server 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809)
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 TERM=xterm-256color
 PATH=(custom, no user)
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.NetworkManager.dispatcher.d.hook-network-manager: [deleted]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Falcon (falcojr)]

See https://github.com/canonical/cloud-init/pull/4264

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3~4ga31370af-0ubuntu1

---------------
cloud-init (23.3~4ga31370af-0ubuntu1) mantic; urgency=medium

  * Upstream snapshot based on upstream/main at a31370af.
    - Bugs fixed in this snapshot: (LP: #1923363, #2028562, #2028784)
  * d/cloud-init.maintscript: Remove the unused hook-network-manager
    conffile. (LP: #2027861)
  * d/control: Add python3-passlib as needed for testing

 -- James Falcon <email address hidden> Mon, 31 Jul 2023 11:38:42 -0500

[Tired Sysadmin (wearyofallthiscrap)]

So perhaps I'm missing something, but that 23.3 package doesn't seem to be showing up on Jammy Jellyfish; an upgrade today got the same 23.2.1-0ubuntu0~22.04.1 against which this bug was originally filed.

Do the fixes normally take this long to propagate? (The breaking change was important enough to go to an LTS release, so presumably the fixing change qualifies also.)

[James Falcon (falcojr)]

23.3 is currently still in development. This fix has only made it to the development release of Ubuntu so far (Mantic). 23.3 should be released upstream soon (likely sometime this week), and then released to the currently supported releases of Ubuntu over the week or two after that.

[Tired Sysadmin (wearyofallthiscrap)]

Status a month later:

Reading the pull link in comment #3 and the conversation linked to from that in turn: https://github.com/canonical/cloud-init/pull/4298 It looks like this fix got reverted because of personnel policy (user committing the fix wasn't allowed to make packaging changes at the time, or similar). So even though 23.3 finally got released two weeks ago, this fix isn't listed in the release notes for https://github.com/canonical/cloud-init/releases/tag/23.3 and doesn't seem to have made it in.

So, other sysadmins: for now you'll still need to clean up the directories manually to avoid the otherwise harmless warnings from dpkg. This fix will be committed eventually once Canonical's paperwork catches up.

[James Falcon (falcojr)]

Tired Sysadmin, what you say isn't true. That PR you reference was an attempt for me to get an upload to happen a day or two earlier, but I realized that it wouldn't work.

The change isn't in upstream because it doesn't affect upstream. It is an Ubuntu-packaging specific issue, so it only exists within Ubuntu packaging. It will be included as part of the Ubuntu packaging when 23.3 is released to our supported series. 23.3 hasn't made it to supported series as that requires a separate process. See https://wiki.ubuntu.com/StableReleaseUpdates and https://wiki.ubuntu.com/CloudinitUpdates for details. The 23.3 release with the packaging change you're looking for has been uploaded as is currently waiting on review to be accepted into the proposed pocket. Once it is accepted, it will be available for testing. Once testing and requisite soak time has completed, it will be released to -updates accordingly.

[Tired Sysadmin (wearyofallthiscrap)]

I misunderstood what you meant in comment #6. It sounded to me like it was going into upstream. My bad.

I've seen the StableReleaseUpdates page, but the simple fact that little bugs like this one and 1803601, to pick examples at random, get introduced into LTS releases has my managers questioning the effectiveness of that policy in actual practice. :-) Obviously they're not major breakage, but when the log analyzer is flagging error messages from a "safe" upgrade to a major package, the chiefs start asking why the Ubuntu maintainers aren't seeing those errors during testing. I keep explaining how minor the warnings are, but bugs in LTS certainly don't make my job day easier, heheh, and how long it takes to return to normal-LTS-smoothness is frustrating.

[Steve Langasek (vorlon)]

Hello Reinhard, or anyone else affected,

Accepted cloud-init into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.1-0ubuntu1~23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Reinhard, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.1-0ubuntu1~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Reinhard, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.1-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Falcon (falcojr)]

Verifications for focal, jammy, and lunar have been uploaded.

Warning is still present on upgrade, but file is removed by the end of the process.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.1-0ubuntu1~23.04.1

---------------
cloud-init (23.3.1-0ubuntu1~23.04.1) lunar; urgency=medium

  * Upstream snapshot based on upstream/main at ee9078a7.
  * d/cloud-init.maintscript: Remove the unused hook-network-manager
    conffile. (LP: #2027861)
  * d/patches/retain-old-groups.patch:
    - Retain original groups in cloud.cfg.tmpl
  * d/control: Add python3-passlib as needed for testing
  * d/cloud-init.templates: enable Akamai by default.
    Add Akamai to the default templates to allow datasource discovery.
  * d/po/templates.pot: refresh with debconf-updatepo
  * Upstream snapshot based on 23.3.1. (LP: #2033310).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.1/ChangeLog

 -- James Falcon <email address hidden> Wed, 06 Sep 2023 11:28:36 -0500

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.1-0ubuntu1~22.04.1

---------------
cloud-init (23.3.1-0ubuntu1~22.04.1) jammy; urgency=medium

  * d/cloud-init.maintscript: Remove the unused hook-network-manager
    conffile. (LP: #2027861)
  * d/patches/retain-old-groups.patch:
    - Retain original groups in cloud.cfg.tmpl
  * Upstream snapshot based on upstream/main at 5509b790.
  * d/control: Add python3-passlib as needed for testing
  * d/cloud-init.templates: enable Akamai by default.
    Add Akamai to the default templates to allow datasource discovery.
  * d/po/templates.pot: refresh with debconf-updatepo
  * Upstream snapshot based on 23.3.1. (LP: #2033310).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.1/ChangeLog

 -- James Falcon <email address hidden> Wed, 06 Sep 2023 12:09:54 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.1-0ubuntu1~20.04.1

---------------
cloud-init (23.3.1-0ubuntu1~20.04.1) focal; urgency=medium

  * Upstream snapshot based on upstream/main at ee9078a7.
  * d/cloud-init.maintscript: Remove the unused hook-network-manager
    conffile. (LP: #2027861)
  * d/patches/retain-old-groups.patch:
    - Retain original groups in cloud.cfg.tmpl
  * d/control: Add python3-passlib as needed for testing
  * d/cloud-init.templates: enable Akamai by default.
    Add Akamai to the default templates to allow datasource discovery.
  * d/po/templates.pot: refresh with debconf-updatepo
  * Upstream snapshot based on 23.3.1. (LP: #2033310).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.1/ChangeLog

 -- James Falcon <email address hidden> Wed, 06 Sep 2023 12:11:17 -0500