cloud-init 23.3.1 provides invalid param --break-system-package to pip

[Chad Smith]

[Impact]

In cloud-init 23.3.1, when cloud-config directives specify that ansible is installed via `install_method: pip` cloud-init leaves ansible uninstalled/configured due to a traceback from pip because cloud-init incorrectly provides the unsupported pip parameter `--break-system-packages` on Focal and Jammy.

This issue was already addressed in upstream commit [1] by only providing the `--break-system-packages` parameter to pip in environments/distributions that have configured python with the marker file EXTERNALLY_MANAGED.

[1] upstream fix https://github.com/canonical/cloud-init/commit/13938f749f7393eaa47a9d5ddd286e675ff2944d

[Test Case]
Use the following script:

#!/bin/bash

for RELEASE in focal jammy lunar mantic; do
  if [ "$RELEASE" = "focal" -o "$RELEASE" = "jammy" ]; then
   echo "Expect integration test FAILURE on $RELEASE with daily images"
   # Run current integration tests are expected to fail path against daily images
    CLOUD_INIT_CLOUD_INIT_SOURCE=NONE CLOUD_INIT_OS_IMAGE=$RELEASE tox -e integration-tests tests/integration_tests/modules/test_ansible.py::test_ansible_pull_pip
  else
   echo "Expect integration test SUCCESS on $RELEASE with daily images"
   # Run current integration tests are expected to fail path against daily images
    CLOUD_INIT_CLOUD_INIT_SOURCE=NONE CLOUD_INIT_OS_IMAGE=$RELEASE tox -e integration-tests tests/integration_tests/modules/test_ansible.py::test_ansible_pull_pip
  fi

   echo "Expect integration test SUCCESS on $RELEASE with SRU version cloud-init 23.3.3"
  # Run current integration tests with this proposed SRU upload from ppa:cloud-init-dev/proposed which will succeed
  CLOUD_INIT_OS_IMAGE=$RELEASE CLOUD_INIT_CLOUD_INIT_SOURCE=ppa:cloud-init-dev/proposed tox -e integration-tests tests/integration_tests/modules/test_ansible.py::test_ansible_pull_pip
done

[Regression Potential]
This is resolving a bug that broke/regressed only environments with have pip version < 23.0.1 and provide the following custom user-data during VM launch:

ansible:
  install_method: pip
   ...

It is a very specific fix for a targeted use-case and will not regress typical deployments using cloud-init

[Other info]

- Community comment filed against upstream PR indicating this problem
https://github.com/canonical/cloud-init/commit/b417b21816827f0b1324cf0df633487c43b15dad#commitcomment-130714494
- Original issue filed against integration test failures https://github.com/canonical/cloud-init/issues/4244

[Chad Smith (chad.smith)]

This bug is believed to be fixed in Ubuntu Nobel in cloud-init version 23.4~3g0cb0b80f-0ubuntu1 and later.
If it is still a problem on later releases, please feel free to reopen this bug.

[Andreas Hasenack (ahasenack)]

Be sure to run the test with the packages from proposed, and not a ppa, like is currently stated in the test plan.

[Andreas Hasenack (ahasenack)]

Hello Chad, or anyone else affected,

Accepted cloud-init into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Chad, or anyone else affected,

Accepted cloud-init into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

As a general remark, for hotfixes such as this, I would have expected the d/changelog entry to contain a brief description of the fix, instead of pointing at the upstream release notes (which contain a brief description of the sole change in the new upstream release). It's a level of indirection I wasn't expecting for this case.

[Andreas Hasenack (ahasenack)]

Hello Chad, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Chad, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Falcon (falcojr)]

Output came from running the script describe in the SRU test case.

Note the 2 expected failures and the 6 passes.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.3-0ubuntu0~23.10.1

---------------
cloud-init (23.3.3-0ubuntu0~23.10.1) mantic; urgency=medium

  * Upstream snapshot based on 23.3.3. (LP: #2040291).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.3/ChangeLog

cloud-init (23.3.2-0ubuntu0~23.10.1) mantic; urgency=medium

  [ Brett Holman ]
  * d/p/do-not-block-user-login.patch:
    - Revert behavior, allow user login after cloud-init stage (LP: #2039505)
  * cherry-pick 2ab1f340: fix(cc_apt_configure): avoid unneeded call to
    apt-get update (LP: #2039621)
  * Upstream snapshot based on 23.3.2. (LP: #2039453).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.2/ChangeLog

 -- James Falcon <email address hidden> Tue, 24 Oct 2023 10:43:22 -0500

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.3-0ubuntu0~23.04.1

---------------
cloud-init (23.3.3-0ubuntu0~23.04.1) lunar; urgency=medium

  * Upstream snapshot based on 23.3.3. (LP: #2040291).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.3/ChangeLog

cloud-init (23.3.2-0ubuntu0~23.04.1) lunar; urgency=medium

  * d/p/do-not-block-user-login.patch:
    - Revert behavior, allow user login after cloud-init stage (LP: #2039505)
  * Upstream snapshot based on 23.3.2. (LP: #2039453).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.2/ChangeLog

 -- James Falcon <email address hidden> Tue, 24 Oct 2023 10:47:46 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.3-0ubuntu0~22.04.1

---------------
cloud-init (23.3.3-0ubuntu0~22.04.1) jammy; urgency=medium

  * Upstream snapshot based on 23.3.3. (LP: #2040291).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.3/ChangeLog

cloud-init (23.3.2-0ubuntu0~22.04.1) jammy; urgency=medium

  * d/p/do-not-block-user-login.patch:
    - Revert behavior, allow user login after cloud-init stage (LP: #2039505)
  * Upstream snapshot based on 23.3.2. (LP: #2039453).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.2/ChangeLog

 -- James Falcon <email address hidden> Tue, 24 Oct 2023 10:50:55 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 23.3.3-0ubuntu0~20.04.1

---------------
cloud-init (23.3.3-0ubuntu0~20.04.1) focal; urgency=medium

  * Upstream snapshot based on 23.3.3. (LP: #2040291).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.3/ChangeLog

cloud-init (23.3.2-0ubuntu0~20.04.1) focal; urgency=medium

  * d/p/do-not-block-user-login.patch:
    - Revert behavior, allow user login after cloud-init stage (LP: #2039505)
  * Upstream snapshot based on 23.3.2. (LP: #2039453).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.3.2/ChangeLog

 -- James Falcon <email address hidden> Tue, 24 Oct 2023 10:59:56 -0500