Unnecessary call to 'apt update' on empty user-data

[Chad Smith]

=== Begin SRU Template ===
[Impact]
Upstream commit 015543d304[1] introduced the ability to install missing dependencies (software-properties-common or gnupg) when running on minimal images with custom cloud-config that contains APT configuration that requires interaction with those tools.

The commit also introduced a default case on first boot where, regardless of whether apt configuration is present, apt update will be called.
This adds an initial boot delay of around 2 seconds and apt update should be avoided unless packages are going to be installed or `package_update: true` is provided in cloud-config user-data.

The fix is to avoid calling apt-get update in default case when no missing dependencies are required for installation.

[Test Case]
* launch daily mantic images with no user-data and confirm apt-get update is called by cloud-init on first boot
* upgrade cloud-init in the mantic VM to the proposed SRU cloud-init deb
* cloud-init clean --logs --reboot to re-run upgraded cloud-init across reboot
* assert that newly booted upgraded VM no longer calls apt-get update

test procedure:

cat > setup-proposed.sh <<EOF
#/bin/bash
mirror=http://archive.ubuntu.com/ubuntu
echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
apt-get update -q;
apt-get install -t \$(lsb_release -sc)-proposed -qy cloud-init;
EOF

cat sru-2039621.sh <<EOF
#!/bin/bash
for release in mantic; do
 VM=sru-$release-2039621
 lxc launch ubuntu-daily:$release $VM
 lxc exec $VM -- cloud-init status --wait --long

 APT_UPDATE_CALLS=$(lxc exec $VM -- grep apt-get /var/log/cloud-init.log)
 [ -n "$APT_UPDATE_CALLS" ] || echo "FAILURE: did not find expected apt-get update calls on empty user-data" && echo "SUCCESS: found apt-get update calls $APT_UPDATE_CALLS"
 echo " --- Assert apt-get update called across clean reboot on released pkg"
 lxc exec $VM -- cloud-init clean --logs --reboot
 sleep 5
 lxc exec $VM -- cloud-init status --wait --long
 APT_UPDATE_CALLS=$(lxc exec $VM -- grep apt-get /var/log/cloud-init.log)
 [ -n "$APT_UPDATE_CALLS" ] || echo "FAILURE: did not find expected apt-get update calls on empty user-data" && echo "SUCCESS: found reproducer: $(echo $APT_UPDATE_CALLS | wc -l ) apt-get update call across clean reboot for published cloud-init"
 echo "---- install proposed cloud-init"
 lxc file push setup-proposed.sh $VM/
 lxc exec $VM -- bash /setup-proposed.sh | grep cloud-init
 RELEASE_DEB=$(pull-ppa-debs ppa:cloud-init-dev/proposed cloud-init ${release} | awk '/.deb/{print $NF}')
 lxc file push $RELEASE_DEB $VM/
 lxc exec $VM -- dpkg -i /$RELEASE_DEB
 echo " --- Assert apt-get update NOT called across clean reboot on proposed pkg"
 lxc exec $VM -- cloud-init clean --logs --reboot
 sleep 5
 lxc exec $VM -- cloud-init status --wait --long
 APT_UPDATE_CALLS=$(lxc exec $VM -- grep apt-get /var/log/cloud-init.log)
 [ -z "$APT_UPDATE_CALLS" ] || echo "FAILURE: found $(echo $APT_UPDATE_CALLS | wc -l ) unexpected apt-get update calls per boot: $APT_UPDATE_CALLS" && echo "SUCCESS: found NO apt-get update logs per boot on empty user-data"
 lxc delete $VM --force
done
EOF

bash sru-2039621.sh # look for SUCCESS|FAIL messages

[Regression Potential]
If done wrong, minimal images may not attempt to install gnupg or software-properties-common packages when cloud-config user-data specifies apt configuration that requires setting up PPAs or importing gpg keys.
This regression test is already covered by https://github.com/canonical/cloud-init/blob/main/tests/integration_tests/modules/test_apt_functionality.py#L419-L447

[Other info]
Existing integration test for installing missing dependencies https://github.com/canonical/cloud-init/blob/main/tests/integration_tests/modules/test_apt_functionality.py#L419-L447

[Original description]

Upstream commit 015543d304[1] introduced the ability to install missing dependencies (software-properties-common or gnupg) when running on minimal images with custom cloud-config that contains APT configuration that requires interaction with those tools.

The commit also introduced a default case on first boot where, regardless of whether apt configuration is present, apt update will be called.
This adds an initial boot delay of around 2 seconds and apt update should be avoided unless packages are going to be installed or `package_update: true` is provided in cloud-config user-data.

[1] https://github.com/canonical/cloud-init/commit/015543d304

The resolution here is to avoid calling the method cloud.distro.install_packages(missing_packages) when no packages need to be installed because the side-effect of this install_packages call is a run of 'apt update'.

[Alberto Contreras (aciba)]

Upstream PR: https://github.com/canonical/cloud-init/pull/4519

[Chad Smith (chad.smith)]

Marking the tasks for Lunar, Jammy and Focal as invalid because this upstream commit only released to mantic and was not SRU'd back to L, J or F series.

[Chad Smith (chad.smith)]

This fix landed in main in https://github.com/canonical/cloud-init/commit/2ab1f3404e1f57ec16e49b8e95ccd1e6e1ea49a7 and was subsequently released to Nobel as cloud-init 23.4~3g0cb0b80f-0ubuntu1

[Andreas Hasenack (ahasenack)]

Just to be sure, in the test plan, this step:

* cloud-init clean --logs --reboot to re-run upgraded cloud-init across reboot

Could you amend it to run that once with the non-proposed package, to confirm that after the clean log, it will call apt-get update again? This just to make sure that "clean" really resets things, and then we can be sure that when the proposed package is installed, it's the fix that is making "apt-get update" not being called again, and not the clean command.

[Chad Smith (chad.smith)]

Thanks Andreas. Updated test procedure to do a pre-upgrade clean --logs --reboot test and assert that apt-get update is called across every clean boot on released version of cloud-init

[Andreas Hasenack (ahasenack)]

Hello Chad, or anyone else affected,

Accepted cloud-init into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chad Smith (chad.smith)]

Asserted current mantic cloud-init 23.3.1-0ubuntu2 reproduces the errant apt-get update calls across clean reboot via: cloud-init clean --logs --reboot. After -proposed upgrade to 23.3.3-0ubuntu0~23.10.1 apt-get update is no longer called automatically on every clean boot.

csmith@downtown:/tmp$ bash ./sru-2039621.sh
Creating sru-mantic-2039621
Starting sru-mantic-2039621
....................................................................
status: done
boot_status_code: enabled-by-generator
last_update: Fri, 17 Nov 2023 16:38:03 +0000
detail:
DataSourceLXD
SUCCESS: found apt-get update calls 2023-11-17 16:37:52,929 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update'] with allowed return codes [0] (shell=False, capture=False)
2023-11-17 16:38:02,400 - util.py[DEBUG]: apt-update [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet update] took 9.471 seconds
2023-11-17 16:38:02,400 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install'] with allowed return codes [0] (shell=False, capture=False)
2023-11-17 16:38:02,825 - util.py[DEBUG]: apt-install [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install] took 0.424 seconds
 --- Assert apt-get update called across clean reboot on released pkg
................
status: done
boot_status_code: enabled-by-generator
last_update: Fri, 17 Nov 2023 16:38:13 +0000
detail:
DataSourceLXD
SUCCESS: found reproducer: 1 apt-get update call across clean reboot for published cloud-init
---- install proposed cloud-init
  cloud-init
Get:1 http://archive.ubuntu.com/ubuntu mantic-proposed/main amd64 cloud-init all 23.3.3-0ubuntu0~23.10.1 [578 kB]
dpkg-preconfigure: unable to re-open stdin: No such file or directory
Preparing to unpack .../cloud-init_23.3.3-0ubuntu0~23.10.1_all.deb ...
Unpacking cloud-init (23.3.3-0ubuntu0~23.10.1) over (23.3.1-0ubuntu2) ...
Setting up cloud-init (23.3.3-0ubuntu0~23.10.1) ...
(Reading database ... 33546 files and directories currently installed.)
Preparing to unpack .../cloud-init_23.3.3-0ubuntu0~23.10.1_all.deb ...
Unpacking cloud-init (23.3.3-0ubuntu0~23.10.1) over (23.3.3-0ubuntu0~23.10.1) ...
Setting up cloud-init (23.3.3-0ubuntu0~23.10.1) ...
Processing triggers for rsyslog (8.2306.0-2ubuntu2) ...
Processing triggers for man-db (2.11.2-3) ...
 --- Assert apt-get update NOT called across clean reboot on proposed pkg
...
status: done
boot_status_code: enabled-by-generator
last_update: Fri, 17 Nov 2023 16:38:43 +0000
detail:
DataSourceLXD
SUCCESS: found NO apt-get update logs per boot on empty user-data

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.3-0ubuntu0~23.10.1

---------------
cloud-init (23.3.3-0ubuntu0~23.10.1) mantic; urgency=medium

  * Upstream snapshot based on 23.3.3. (LP: #2040291).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.3/ChangeLog

cloud-init (23.3.2-0ubuntu0~23.10.1) mantic; urgency=medium

  [ Brett Holman ]
  * d/p/do-not-block-user-login.patch:
    - Revert behavior, allow user login after cloud-init stage (LP: #2039505)
  * cherry-pick 2ab1f340: fix(cc_apt_configure): avoid unneeded call to
    apt-get update (LP: #2039621)
  * Upstream snapshot based on 23.3.2. (LP: #2039453).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.2/ChangeLog

 -- James Falcon <email address hidden> Tue, 24 Oct 2023 10:43:22 -0500

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.