cloud-init-23.4 cannot read "- Azure" datasource_list format

[Alberto Contreras]

[ Impact ]

When an invalid configuration in /etc/cloud/cloud.cfg.d/ includes an invalid (empty) datasource_list, ds-identify will identify the wrong datasource.

While the datasource_list is invalid, it does represent a regression because previous ds-identify behavior was to silently ignore the invalid configuration and now it needlessly prevents cloud-init from running rather than identifying the correct datasource like it used to.

[ Test Plan ]

1. Create a VM on Azure, change the /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg to:

datasource:
  Azure:
    apply_network_config: false
datasource_list:
 \- Azure
2. Clean cloud-init and reboot VM

 $sudo cloud-init clean
 $sudo reboot

Expected results
cloud-init service can start successfully.

Bug results
cloud-init service fails to start.

[ Where problems could occur ]

This is a revert of a commit added this cycle, so it's possible we could bring back any other undiscovered bugs we didn't know about in the old code.

[ Other Info ]

Upstream bug: https://github.com/canonical/cloud-init/issues/4794

This issue was introduced as a fix for https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2030729 in https://github.com/canonical/cloud-init/pull/4327

PR reverting and fixing the issue: https://github.com/canonical/cloud-init/pull/4797

[ Original Description ]

In Image Builder images, the datasource_list in /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg is:

datasource_list:
 \- Azure

which cannot be parsed by cloud-init.

2024-01-18 10:27:12,433 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.rhel.Distro'>
2024-01-18 10:27:12,441 - _{_}init{_}_.py[DEBUG]: Looking for data source in: [\\{'datasource_list': None}, 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM', 'NETWORK']
2024-01-18 10:27:12,441 - util.py[WARNING]: failed stage init
2024-01-18 10:27:12,441 - util.py[DEBUG]: failed stage init
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 781, in status_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 394, in main_init
    init.fetch(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 493, in fetch
    return self._get_data_source(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 367, in _get_data_source
    self.reporter,
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1001, in find_source
    ds_list = list_sources(cfg_list, ds_deps, pkg_list)
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1047, in list_sources
    ds_name = importer.match_case_insensitive_module_name(ds)
  File "/usr/lib/python3.6/site-packages/cloudinit/importer.py", line 40, in match_case_insensitive_module_name
    if "nocloud-net" == mod_name.lower():
AttributeError: 'dict' object has no attribute 'lower'

## Steps to reproduce the problem
1. Create a VM on Azure, change the /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg to:

datasource:
  Azure:
    apply_network_config: false
datasource_list:
 \- Azure
2. Clean cloud-init and reboot VM

 $sudo cloud-init clean
 $sudo reboot

Expected results
cloud-init service can start successfully.

Actual results
cloud-init service fails to start(logs are in the description)
If change the

datasource_list:
 \- Azure
to:

datasource_list: [ Azure ]
Then cloud-init clean && reboot, the cloud-init service and start successfully.

## Environment details
- Cloud-init version: cloud-init-23.4
- Operating System Distribution: RHEL
- Cloud provider, platform or installer type: Azure

## cloud-init logs
1. $ sudo cat /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg
 datasource_list:
 \- Azure

2. cloud-init.log

2024-01-18 10:27:12,433 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.rhel.Distro'>
2024-01-18 10:27:12,441 - _{_}init{_}_.py[DEBUG]: Looking for data source in: [\\{'datasource_list': None}, 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM', 'NETWORK']
2024-01-18 10:27:12,441 - util.py[WARNING]: failed stage init
2024-01-18 10:27:12,441 - util.py[DEBUG]: failed stage init
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 781, in status_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 394, in main_init
    init.fetch(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 493, in fetch
    return self._get_data_source(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 367, in _get_data_source
    self.reporter,
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1001, in find_source
    ds_list = list_sources(cfg_list, ds_deps, pkg_list)
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1047, in list_sources
    ds_name = importer.match_case_insensitive_module_name(ds)
  File "/usr/lib/python3.6/site-packages/cloudinit/importer.py", line 40, in match_case_insensitive_module_name
    if "nocloud-net" == mod_name.lower():
AttributeError: 'dict' object has no attribute 'lower'

## Additional info:
1. No such issue in cloud-init-23.1.1, just met this issue after rebase to 23.4, so could you please help to address which commit did this change between 23.1.1 and 23.4?
2. RHEL was using ds-identify before 23.1.1
3. I am afraid users who were using the format "- Azure" in config file will meet this issue after upgrade, so could you please help to check if there is a way to be compatible with this format? Thanks

---

Upstream bug: https://github.com/canonical/cloud-init/issues/4794

This issue was introduced as a fix for https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2030729 in https://github.com/canonical/cloud-init/pull/4327

PR reverting and fixing the issue: https://github.com/canonical/cloud-init/pull/4797

[Chad Smith (chad.smith)]

Fix committed upstream per https://github.com/canonical/cloud-init/commit/8ff94fe9493ad88344eb8bbf2f023c6ba2db5206

[Chad Smith (chad.smith)]

published to Ubuntu Noble: cloud-init 24.1~4gd9677655-0ubuntu1 (Accepted)

[Andreas Hasenack (ahasenack)]

Hello Alberto, or anyone else affected,

Accepted cloud-init into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.2-0ubuntu0~23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Alberto, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.2-0ubuntu0~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Alberto, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.2-0ubuntu0~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Alberto, or anyone else affected,

Accepted cloud-init into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.3-0ubuntu0~23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Alberto, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.3-0ubuntu0~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Alberto, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.3-0ubuntu0~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alberto Contreras (aciba)]

I have verified that the bug is not reproducible with 23.4.3-0ubuntu0~23.10.1, 23.4.3-0ubuntu0~22.04.1 and 23.4.3-0ubuntu0~20.04.1. Attached test script and logs.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.4.3-0ubuntu0~23.10.1

---------------
cloud-init (23.4.3-0ubuntu0~23.10.1) mantic; urgency=medium

  * Upstream snapshot based on 23.4.3. (LP: #2046483).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.3/ChangeLog

cloud-init (23.4.2-0ubuntu0~23.10.1) mantic; urgency=medium

  * Upstream snapshot based on 23.4.2. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.2/ChangeLog
    - Bugs fixed in this snapshot: (LP: #2051147)

cloud-init (23.4.1-0ubuntu1~23.10.2) mantic; urgency=medium

  * d/p/status-retain-recoverable-error-exit-code.patch:
    Retain exit code in cloud-init status for recoverable errors.
    (LP: #2048522).

cloud-init (23.4.1-0ubuntu1~23.10.1) mantic; urgency=medium

  * d/p/retain-apt-pre-deb822.patch:
    - Disable apt source list generation with DEB822 style
  * d/p/do-not-block-user-login.patch:
    - revert redacted patch content introduced in 23.4-0
  * refresh patches:
    - d/p/status-do-not-remove-duplicated-data.patch
  * d/changelog: amend 23.4-0 refresh patches and dropped cherry-picks entry
  * Upstream snapshot based on 23.4.1. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.1/ChangeLog

cloud-init (23.4-0ubuntu1~23.10.1) mantic; urgency=medium

  * d/p/status-do-not-remove-duplicated-data.patch:
    - Revert behavior downstream, leave duplicate data
  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * refresh patches:
    - d/p/do-not-block-user-login.patch
  * drop the following cherry-picks now included:
    - cpick-0d9f149a-Pytestify-apt-config-test-modules-4424
    - cpick-5023e9f9-Refactor-test_apt_source_v1.py-to-use-pytest-4427
    - cpick-e9cdd7e3-Install-gnupg-if-gpg-not-found-4431
    - cpick-015543d3-apt-install-software-properties-common-when-absent-but
    - cpick-2ab1f340-fix-cc_apt_configure-avoid-unneeded-call-to-apt-install
  * Upstream snapshot based on 23.4. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4/ChangeLog

 -- James Falcon <email address hidden> Fri, 02 Feb 2024 16:00:04 -0600

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.4.3-0ubuntu0~22.04.1

---------------
cloud-init (23.4.3-0ubuntu0~22.04.1) jammy; urgency=medium

  * Upstream snapshot based on 23.4.3. (LP: #2046483).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.3/ChangeLog

cloud-init (23.4.2-0ubuntu0~22.04.1) jammy; urgency=medium

  * Upstream snapshot based on 23.4.2. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.2/ChangeLog
    - Bugs fixed in this snapshot: (LP: #2051147)

cloud-init (23.4.1-0ubuntu1~22.04.2) jammy; urgency=medium

  * d/p/status-retain-recoverable-error-exit-code.patch:
    Retain exit code in cloud-init status for recoverable errors.
    (LP: #2048522).

cloud-init (23.4.1-0ubuntu1~22.04.1) jammy; urgency=medium

  * d/p/retain-apt-pre-deb822.patch:
    - Disable apt source list generation with DEB822 style
  * refresh patches:
    - d/p/status-do-not-remove-duplicated-data.patch
  * d/changelog: amend 23.4-0 refresh patches entry
  * Upstream snapshot based on 23.4.1. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.1/ChangeLog

cloud-init (23.4-0ubuntu1~22.04.1) jammy; urgency=medium

  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * d/p/status-do-not-remove-duplicated-data.patch:
    - Revert behavior downstream, leave duplicate data
  * d/p/series: bring back retain-old-groups.patch.
    This patch was inadvertently dropped in 5d4a3cf.
  * refresh patches:
    - d/p/do-not-block-user-login.patch
  * Upstream snapshot based on 23.4. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4/ChangeLog

 -- James Falcon <email address hidden> Fri, 02 Feb 2024 15:59:14 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.4.3-0ubuntu0~20.04.1

---------------
cloud-init (23.4.3-0ubuntu0~20.04.1) focal; urgency=medium

  * Upstream snapshot based on 23.4.3. (LP: #2046483).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.3/ChangeLog

cloud-init (23.4.2-0ubuntu0~20.04.1) focal; urgency=medium

  * Upstream snapshot based on 23.4.2. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.2/ChangeLog
    - Bugs fixed in this snapshot: (LP: #2051147)

cloud-init (23.4.1-0ubuntu1~20.04.2) focal; urgency=medium

  * d/p/status-retain-recoverable-error-exit-code.patch:
    Retain exit code in cloud-init status for recoverable errors.
    (LP: #2048522).

cloud-init (23.4.1-0ubuntu1~20.04.1) focal; urgency=medium

  * d/p/retain-apt-pre-deb822.patch:
    - Disable apt source list generation with DEB822 style
  * refresh patches:
    - d/p/status-do-not-remove-duplicated-data.patch
  * d/changelog: amend 23.4-0 refresh patches entry
  * Upstream snapshot based on 23.4.1. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.1/ChangeLog

cloud-init (23.4-0ubuntu1~20.04.1) focal; urgency=medium

  * d/p/status-do-not-remove-duplicated-data.patch:
    - Revert behavior downstream, leave duplicate data
  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * refresh patches:
    - d/p/do-not-block-user-login.patch
    - d/p/netplan99-cannot-use-default.patch
  * Upstream snapshot based on 23.4. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4/ChangeLog

 -- James Falcon <email address hidden> Fri, 02 Feb 2024 16:00:45 -0600