sru cloud-init (23.3 update) Focal, Jammy, and Lunar

[James Falcon]

== Begin SRU Template ==
[Impact]
This release sports both bug-fixes and new features and we would like to
make sure all of our supported customers have access to these
improvements. The notable ones are:
 - Remove python deprecated crypt dependency in favor of
   passlib.hash.bcrypt where present. crypt was used by by NetBSD and
   Azure datasource. Plan is to eventually deprecate passlib dependency too.
 - network: support busybox micro DHCP client (udhcpc) alternative to
   deprecated isc-dhcp-client.
 - Clouds:
   - Ensure Akamai(and Linode) datasource included in default discovery
   - Ec2: support deconding double base64 encoded user-data
   - GCE: improve fallback nic selection ens4 over eth0 as primary/default NIC
 - Distributions:
     - debian/ubuntu: no longer use localectl to set keymap (LP: #2030788)
     - systemd v253++: cloud-init status --wait won't block on spawned daemons
 - Modules:
     - users: support doas/opendoas rules as alternative to sudo
     - PEP 668: pip-installed ansible to support break-system-packages installs
 - tooling: New file /run/cloud-init/combined-cloud-config.json to read all
      merged user-data/config without invoking python.

See the changelog entry below for a full list of changes and bugs.

[Test Case]
The following development and SRU process was followed:
https://wiki.ubuntu.com/CloudinitUpdates

The cloud-init team will be in charge of attaching the artifacts and
console output of the appropriate run to the bug. cloud-init team
members will not mark ‘verification-done’ until this has happened.

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned integration tests are attached to this bug.

[Discussion]

== End SRU Template ==

== Changelog ==
 - Bump pycloudlib to 1!5.1.0 for ec2 mantic daily image support (#4390)
 - Fix cc_keyboard in mantic (LP: #2030788)
 - ec2: initialize get_instance_userdata return value to bytes (#4387)
   [Noah Meyerhans]
 - cc_users_groups: Add doas/opendoas support (#4363) [dermotbradley]
 - Fix pip-managed ansible
 - status: treat SubState=running and MainPID=0 as service exited
 - azure/imds: increase read-timeout to 30s (#4372) [Chris Patterson]
 - collect-logs fix memory usage (SC-1590) (#4289)
   [Alec Warren] (LP: #1980150)
 - cc_mounts: Use fallocate to create swapfile on btrfs (#4369) [王煎饼]
 - Undocument nocloud-net (#4318)
 - feat(akamai): add akamai to settings.py and apport.py (#4370)
 - read-version: fallback to get_version when git describe fails (#4366)
 - apt: fix cloud-init status --wait blocking on systemd v 253 (#4364)
 - integration tests: Pass username to pycloudlib (#4324)
 - Bump pycloudlib to 1!5.1.0 (#4353)
 - cloud.cfg.tmpl: reorganise, minimise/reduce duplication (#4272)
   [dermotbradley]
 - analyze: fix (unexpected) timestamp parsing (#4347) [Mina Galić]
 - cc_growpart: fix tests to run on FreeBSD (#4351) [Mina Galić]
 - subp: Fix spurious test failure on FreeBSD (#4355) [Mina Galić]
 - cmd/clean: fix tests on non-Linux platforms (#4352) [Mina Galić]
 - util: Fix get_proc_ppid() on non-Linux systems (#4348) [Mina Galić]
 - cc_wireguard: make tests pass on FreeBSD (#4346) [Mina Galić]
 - unittests: fix breakage in test_read_cfg_paths_fetches_cached_datasource
   (#4328) [Ani Sinha]
 - Fix test_tools.py collection (#4315)
 - cc_keyboard: add Alpine support (#4278) [dermotbradley]
 - Flake8 fixes (#4340) [Robert Schweikert]
 - cc_mounts: Fix swapfile not working on btrfs (#4319) [王煎饼] (LP: #1884127)
 - ds-identify/CloudStack: $DS_MAYBE if vm running on vmware/xen (#4281)
   [Wei Zhou]
 - ec2: Support double encoded userdata (#4276) [Noah Meyerhans]
 - cc_mounts: xfs is a Linux only FS (#4334) [Mina Galić]
 - tests/net: fix TestGetInterfaces' mock coverage for get_master (#4336)
   [Chris Patterson]
 - change openEuler to openeuler and fix some bugs in openEuler (#4317)
   [sxt1001]
 - Replace flake8 with ruff (#4314)
 - NM renderer: set default IPv6 addr-gen-mode for all interfaces to eui64
   (#4291) [Ani Sinha]
 - cc_ssh_import_id: add Alpine support and add doas support (#4277)
   [dermotbradley]
 - Release 23.2.2 (#4300)
 - sudoers not idempotent (SC-1589) (#4296) [Alec Warren] (LP: #1998539)
 - Added support for Akamai Connected Cloud (formerly Linode) (#4167)
   [Will Smith]
 - Fix reference before assignment (#4292)
 - Overhaul module reference page (#4237) [Sally]
 - replaced spaces with commas for setting passenv (#4269) [Alec Warren]
 - DS VMware: modify a few log level (#4284) [PengpengSun]
 - tools/read-version refactors and unit tests (#4268)
 - Ensure get_features() grabs all features (#4285)
 - Don't always require passlib dependency (#4274)
 - tests: avoid leaks into host system checking of ovs-vsctl cmd (#4275)
 - Fix NoCloud kernel commandline key parsing (#4273)
 - testing: Clear all LRU caches after each test (#4249)
 - Remove the crypt dependency (#2139) [Gonéri Le Bouder]
 - logging: keep current file mode of log file if its stricter than the
   new mode (#4250) [Ani Sinha]
 - Remove default membership in redundant groups (#4258)
   [Dave Jones] (LP: #1923363)
 - doc: improve datasource_creation.rst (#4262)
 - Remove duplicate Integration testing button (#4261) [Rishita Shaw]
 - tools/read-version: fix the tool so that it can handle version parsing
   errors (#4234) [Ani Sinha]
 - net/dhcp: add udhcpc support (#4190) [Jean-François Roche]
 - DS VMware: add i386 arch dir to deployPkg plugin search path
   [PengpengSun]
 - LXD moved from linuxcontainers.org to Canonical [Simon Deziel]
 - cc_mounts.py: Add note about issue with creating mounts inside mounts
   (#4232) [dermotbradley]
 - lxd: install lxd from snap, not deb if absent in image
 - landscape: use landscape-config to write configuration
 - Add deprecation log during init of DataSourceDigitalOcean (#4194)
   [tyb-truth]
 - doc: fix typo on apt.primary.arches (#4238) [Dan Bungert]
 - Inspect systemd state for cloud-init status (#4230)
 - instance-data: add system-info and features to combined-cloud-config
   (#4224)
 - systemd: Block login until config stage completes (#2111) (LP: #2013403)
 - tests: proposed should invoke apt-get install -t=<release>-proposed
   (#4235)
 - cloud.cfg.tmpl: reinstate ca_certs entry (#4236) [dermotbradley]
 - Remove feature flag override ability (#4228)
 - tests: drop stray unrelated file presence test (#4227)
 - Update LXD URL (#4223) [Sally]
 - schema: add network v1 schema definition and validation functions
 - tests: daily PPA for devel series is version 99.daily update tests to
   match (#4225)
 - instance-data: write /run/cloud-init/combined-cloud-config.json
 - mount parse: Fix matching non-existent directories (#4222) [Mina Galić]
 - Specify build-system for pep517 (#4218)
 - Fix network v2 metric rendering (#4220)
 - Migrate content out of FAQ page (SD-1187) (#4205) [Sally]
 - setup: fix generation of init templates (#4209) [Mina Galić]
 - docs: Correct some bootcmd example wording
 - fix changelog
 - Release 23.2.1 (#4207) (LP: #2025180)
 - tests: reboot client to assert x-shellscript-per-boot is triggered
 - nocloud: parse_cmdline no longer detects nocloud-net datasource (#4204)
   (LP: 4203, #2025180)
 - Add docstring and typing to mergemanydict (#4200)
 - BSD: add dsidentify to early startup scripts (#4182) [Mina Galić]
 - handler: report errors on skipped merged cloud-config.txt parts
   (LP: #1999952)
 - Add cloud-init summit writeups (#4179) [Sally]
 - tests: Update test_clean_log for oci (#4187)
 - gce: improve ephemeral fallback NIC selection (CPC-2578) (#4163)
 - tests: pin pytest 7.3.1 to avoid adverse testpaths behavior (#4184)
 - Ephemeral Networking for FreeBSD (#2165) [Mina Galić]
 - Clarify directory syntax for nocloud local filesystem. (#4178)
 - Set default renderer as sysconfig for centos/rhel (#4165) [Ani Sinha]
 - Test static routes and netplan 0.106
 - FreeBSD fix parsing of mount and mount options (#2146) [Mina Galić]
 - test: add tracking bug id (#4164)
 - tests: can't match MAC for LXD container veth due to netplan 0.106
   (#4162)
 - Add kaiwalyakoparkar as a contributor (#4156) [Kaiwalya Koparkar]
 - BSD: remove datasource_list from cloud.cfg template (#4159) [Mina Galić]
 - launching salt-minion in masterless mode (#4110) [Denis Halturin]
 - tools: fix run-container builds for rockylinux/8 git hash mismatch
   (#4161)
 - fix doc lint: spellchecker tripped up (#4160) [Mina Galić]
 - Support Ephemeral Networking for BSD (#2127)
 - Added / fixed support for static routes on OpenBSD and FreeBSD (#2157)
   [Kadir Mueller]
 - cc_rsyslog: Refactor for better multi-platform support (#4119)
   [Mina Galić] (LP: #1798055)
 - tests: fix test_lp1835584 (#4154)
 - cloud.cfg mod names: docs and rename salt_minion and set_password (#4153)
 - tests: apt support for deb822 format .sources files on mantic
 - vultr: remove check_route check (#2151) [Jonas Chevalier]
 - Update SECURITY.md (#4150) [Indrranil Pawar]
 - Update CONTRIBUTING.rst (#4149) [Indrranil Pawar]
 - Update .github-cla-signers (#4151) [Indrranil Pawar]
 - Standardise module names in cloud.cfg.tmpl to only use underscore
   (#4128) [dermotbradley]
 - tests: update test_webhook_reporting
 - Modify PR template so autoclose works
 - doc: add missing semi-colon to nocloud cmdline docs (#4120)
 - .gitignore: extend coverage pattern (#4143) [Mina Galić]

[Steve Langasek (vorlon)]

While it seems like a minor point, I'm not accepting this right off because of the version numbers used. mantic currently has 23.3-0ubuntu1, but the version number uploaded to lunar is 23.3-0ubuntu0~23.04.1. This sorts before 23.3-0ubuntu1 so technically meets the minimum requirements for version ordering in SRU, but it also sorts earlier than an arbitrary number of other package versions not in the archive, such as 23.3-0ubuntu0, 23.3-0ubuntu0+ppa.1, 23.3-0ubuntu1~23.04, etc.

The normal convention here would be to use 23.3-0ubuntu1~23.04.1 as a version number, signalling that this is based on 23.3-0ubuntu1, is a backport to 23.04, and there are no sourceful differences between the two.

That would also simplify handling in debian/cloud-init.maintscript, which has just added the line:

+rm_conffile /etc/NetworkManager/dispatcher.d/hook-network-manager 23.3-0ubuntu0~23.04.1~

... if using a more typical version scheme, you could use '23.3-0ubuntu1~' as the version check, across all releases, and get the intended results.

[Steve Langasek (vorlon)]

> + * d/patches/retain-old-groups.patch:
> + - Retain original groups in cloud.cfg.tmpl

Hmm, I guess I can't argue that you shouldn't be more conservative and retain compatibility, even though we really shouldn't be adding users to those groups :-)

[Steve Langasek (vorlon)]

Hello James, or anyone else affected,

Accepted cloud-init into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.1-0ubuntu1~23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello James, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.1-0ubuntu1~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello James, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.1-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alexander Balderson (asbalderson)]

Hello all,

SQA has tested cloud-init on jammy and focal and didnt run into any issues.

The results are only visible by Canonical employees but,

The Focal testplan can be found at:
https://solutions.qa.canonical.com/releasetests/6e733fdd-def4-4d7a-be92-c3b3a1b32a76/

and Jammy testplan can be found at:
https://solutions.qa.canonical.com/releasetests/95d10707-9a9e-4752-a661-9f64d201e176/

[James Falcon (falcojr)]

Regression testing attached. This includes:
* Cloud-init integration tests on EC2, GCE, Azure, LXD Containers, and LXD VMs for focal, jammy, and lunar

Any failures were transient, pre-existing issues, or test failures and called out in failures.txt or rerun in <job>-rerun.txt file.

[James Falcon (falcojr)]

Curtin SRU test results

[Chad Smith (chad.smith)]

In manual verification cloud-init -proposed of live-server installs on jammy a regression was discovered due to an older version of cloud-init (23.2.2-0ubuntu0~22.04.1) shipped in the subiquity snap being unable to unpickle the datasource cache.

Subiquity team will determine if this should be a blocked for cloud-init SRU or not given their expectation to release subiquity to stable within a week.

https://bugs.launchpad.net/ubuntu/+source/subiquity/+bug/2038375

[Chad Smith (chad.smith)]

Subiquity team has assessed this expected 'regression' and since there are workarounds and an intended release shortly, this does not pose a problem.

https://bugs.launchpad.net/ubuntu/+source/subiquity/+bug/2038375/comments/1
"I suggest not holding cloud-init SRU, as the fix for this issue will be released on or around October-12th.

"

[Andreas Hasenack (ahasenack)]

We discussed this internally, and considering:
- daily LTS installer images are more oriented towards testing, and not production usage (assumption)
- a fix will be published with the mantic release next Thursday (Oct 12th), and daily images built after that will include it
- only the specific autoinstall scenario is affected
- the workaround is to use the published jammy LTS installer release image, and not the dailies

I think it's OK to proceed with this SRU release. I can, however, see that perhaps some users would prefer the daily image because it will mean less updates to apply to the target after installation. But the benefits of the bugs being fixed in this SRU outweigh the specific scenario of autoinstall with daily images not working for about a week, in my opinion. And the workaround for that period of time is to switch to the actual release-quality images, and not the daily ones.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.1-0ubuntu1~23.04.1

---------------
cloud-init (23.3.1-0ubuntu1~23.04.1) lunar; urgency=medium

  * Upstream snapshot based on upstream/main at ee9078a7.
  * d/cloud-init.maintscript: Remove the unused hook-network-manager
    conffile. (LP: #2027861)
  * d/patches/retain-old-groups.patch:
    - Retain original groups in cloud.cfg.tmpl
  * d/control: Add python3-passlib as needed for testing
  * d/cloud-init.templates: enable Akamai by default.
    Add Akamai to the default templates to allow datasource discovery.
  * d/po/templates.pot: refresh with debconf-updatepo
  * Upstream snapshot based on 23.3.1. (LP: #2033310).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.1/ChangeLog

 -- James Falcon <email address hidden> Wed, 06 Sep 2023 11:28:36 -0500

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.1-0ubuntu1~22.04.1

---------------
cloud-init (23.3.1-0ubuntu1~22.04.1) jammy; urgency=medium

  * d/cloud-init.maintscript: Remove the unused hook-network-manager
    conffile. (LP: #2027861)
  * d/patches/retain-old-groups.patch:
    - Retain original groups in cloud.cfg.tmpl
  * Upstream snapshot based on upstream/main at 5509b790.
  * d/control: Add python3-passlib as needed for testing
  * d/cloud-init.templates: enable Akamai by default.
    Add Akamai to the default templates to allow datasource discovery.
  * d/po/templates.pot: refresh with debconf-updatepo
  * Upstream snapshot based on 23.3.1. (LP: #2033310).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.1/ChangeLog

 -- James Falcon <email address hidden> Wed, 06 Sep 2023 12:09:54 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.1-0ubuntu1~20.04.1

---------------
cloud-init (23.3.1-0ubuntu1~20.04.1) focal; urgency=medium

  * Upstream snapshot based on upstream/main at ee9078a7.
  * d/cloud-init.maintscript: Remove the unused hook-network-manager
    conffile. (LP: #2027861)
  * d/patches/retain-old-groups.patch:
    - Retain original groups in cloud.cfg.tmpl
  * d/control: Add python3-passlib as needed for testing
  * d/cloud-init.templates: enable Akamai by default.
    Add Akamai to the default templates to allow datasource discovery.
  * d/po/templates.pot: refresh with debconf-updatepo
  * Upstream snapshot based on 23.3.1. (LP: #2033310).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.1/ChangeLog

 -- James Falcon <email address hidden> Wed, 06 Sep 2023 12:11:17 -0500