[SRU] cloud-init archive template should match Ubuntu Server

[Ben Howard]

[SRU Justification]
Ubuntu Cloud Images are inconsistent with desktop and bare-metal server installations since backports, restricted and multiverse are not enabled. This is effected via cloud-init that uses a template to select an in-cloud archive.

[FIX] Make the cloud-init template match that of Ubuntu-server.

[REGRESION] The potential for regression is low. However, all users will experience slower fetch times on apt-get updates especially on slower or high latency networks.

[TEST]
1. Build image from -proposed
2. Boot up image
3. Confirm that "sudo apt-get update" pulls in backports, restricted and multiverse.

Backports are currently not enabled in the cloud-init template. This is needed in order to get the backport kernels on cloud images.

Related bugs:
 * bug 997371: Create command to add "multiverse" and "-backports" to apt sources
 * bug 1513529: cloud image built-in /etc/apt/sources.list needs updating

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1177432/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Scott Moser (smoser)]

== summary ==
fix this problem by creating tools to manage /etc/apt/sources.list

== details ==
I realize that cloud-init is just kind of caught in the middle here because of the template that it has, and the fact that that template has not been updated to keep in sync with changes made elsewhere to the default sources list.

I personally prefer the smaller sources.list that is present in the cloud images. For reference, the differences between a server install of 13.04 cd and cloud-init template at the moment are:
 - cloud-init: (12 total entries, 2 x 3 x 2)
    {deb, deb-src} $release{"",-updates,-security} {main, universe}
 - install cd: (32 total entries, 2 x 4 x 4)
    {deb, deb-src} $release{"",-updates,-security,-backports} {main,restricted,universe,multiverse}

Ignoring multiarch, and translations, which are also multipliers, That means 20 more GETs every time 'apt-get update' is done. That is nothing to be scoffed at.

I put a gist at https://gist.github.com/smoser/5586288 which uses chdist to easily time runs of 'apt-get update'. The differences are very real.
In quick tests on raring, I was seeing a refresh taking 20 seconds for "full list" versus 8 seconds for cloud-images list against a locally warmed cache.

This is partially because apt is doing the GET serially, but even the cpu decompression does take time.

My basic issue boils down to this:
 * I expect that when people launch an instance, they immediately install software on it. In order to do that, they need to run 'apt-get update'.
 * If apt-get update goes from taking 8 seconds to 20 seconds, that is a significant increase in total time to functional system.

I'd prefer that we solve this *not* by making cloud-init have a large amount of additional sources. If I were to suggest making a change, it would be to remove deb-src entries from /etc/apt/sources.list to make it smaller.

My suggestion is that we fix this problem by adding tools that manipulate /etc/apt/sources.list or sources.list.d similar to 'apt-add-repository', but that support aliases like 'backports', 'partner', 'multiverse'.
Example usage:
 - aptsources --no-source set main universe
 - aptsources remove multiverse

Cloud-init can then expose such simple instructions in its cloud-config syntax.

[Scott Moser (smoser)]

2 comments:
 a.) bug 997371 is now fixed, so you should be able to do: apt-add-repository multiverse
 b.) some relevant discussion at https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2013-May/014503.html

[Stéphane Graber (stgraber)]

Just ran into this issue again today when testing the LXD backport.

This is pretty annoying as it's causing a completely different user experience for cloud users compared to regular distro users.

Also, at least in trusty, apt-add-repository doesn't know how to add backports, so a user will need to manually update /etc/apt/sources.list and then run apt-get update which makes installing backports significantly harder than it should be.

[Dustin Kirkland  (kirkland)]

This is bad. Please fix this asap.

[Antonio Rosales (arosales)]

Ben,

To confirm the behavior, will the default be kept for selecting back-ported packages? Specifically, in order to install a back-ported image a cloud-image user would have to be explicit in choosing to install from back-port, ie
  $ sudo apt-get install <package-name>/trusty-backports

ref = https://help.ubuntu.com/community/UbuntuBackports

[Antonio Rosales (arosales)]

--- Correction ---

Ben,

To confirm the behavior, will the default be kept for selecting back-ported packages? Specifically, in order to install a back-ported _package_ a cloud-image user would have to be explicit in choosing to install from back-port, ie
  $ sudo apt-get install <package-name>/trusty-backports

ref = https://help.ubuntu.com/community/UbuntuBackports

[Ben Howard (darkmuggle-deactivatedaccount)]

In re to #7/#8, the handling of selections from backports will _NOT_ be changed.

Users will have to specifically opt-in to selecting packages.

i.e to get regular ansible:
   $ sudo apt-get install ansible

Whereas are the following examples are valid for pulling from backports
  $ sudo apt-get -t trusty-backports install ansible
  $ sudo apt-get install lxd/trusty-backports

[Antonio Rosales (arosales)]

Re comment 9, thanks for confirming @utlemming.

-Antonio

[Scott Moser (smoser)]

I've committed the change to trunk that actually puts the template now inline with what is provided by 14.04 and 15.10 server iso installs (as seen in those attachments above).

I've also added tools to https://gist.github.com/smoser/5586288 and some more information. This illustrates well my desire for '--if-necessary' as requested in bug 1429285.

To summarize the changes below. That is roughly 450k additional download data (compressed size). Most notable is a stark doubling of GETs from 27 to 58:

    ## old cloud sources.list after "apt-get clean"
    $ ./summarize-http out.d/cloud/clean/update.err
    ...
    data bytes: 14431826
    total GETS: 27
    200: 16
    404: 11

    ## old cloud sources.list immediately repeated 'apt-get update'
    $ ./summarize-http out.d/cloud/noclean/update.err
    ...
    data bytes: 0
    total GETS: 27
    304: 16
    404: 11

    ## ISO sources.list after "apt-get clean"
    $ ./summarize-http out.d/cd/clean/update.err
    ...
    data bytes: 14893753
    total GETS: 58
    200: 37
    404: 21

    ## ISO sources.list immediately repeated 'apt-get update'
    $ ./summarize-http out.d/cd/noclean/update.err
    ...
    data bytes: 0
    total GETS: 58
    304: 37
    404: 21

[Chris J Arges (arges)]

Hello Ben, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Ben, or anyone else affected,

Accepted cloud-init into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1091-0ubuntu10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Ben, or anyone else affected,

Accepted cloud-init into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1149-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ben Howard (darkmuggle-deactivatedaccount)]

Marking as verification-failed pending a rebase on the new upstream trunk.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1154-0ubuntu1

---------------
cloud-init (0.7.7~bzr1154-0ubuntu1) xenial; urgency=medium

  * New upstream snapshot.
    * create the same /etc/apt/sources.list that is present in default server
      ISO installs. This change adds restricted, multiverse, and -backports
      (LP: #1177432).

 -- Scott Moser <email address hidden> Thu, 05 Nov 2015 12:10:00 -0500

[Chris J Arges (arges)]

Hello Ben, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Ben, or anyone else affected,

Accepted cloud-init into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1091-0ubuntu11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Ben, or anyone else affected,

Accepted cloud-init into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1149-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ben Howard (darkmuggle-deactivatedaccount)]

Verified changes enable multiverse, restricted and backports in cloud images.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1149-0ubuntu3

---------------
cloud-init (0.7.7~bzr1149-0ubuntu3) wily; urgency=medium

  * d/patches/lp-1177432-same-archives-as-ubuntu-server.patch: use the
    same archive pockets as Ubuntu Server (LP: #1177432).

 -- Ben Howard <email address hidden> Thu, 05 Nov 2015 12:41:19 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1091-0ubuntu11

---------------
cloud-init (0.7.7~bzr1091-0ubuntu11) vivid; urgency=medium

  * d/patches/lp-1177432-same-archives-as-ubuntu-server.patch: use the
    same archive pockets as Ubuntu Server (LP: #1177432).

 -- Ben Howard <email address hidden> Thu, 05 Nov 2015 12:36:21 -0700

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.14

---------------
cloud-init (0.7.5-0ubuntu1.14) trusty; urgency=medium

  * d/patches/lp-1177432-same-archives-as-ubuntu-server.patch: use the
    same archive pockets as Ubuntu Server (LP: #1177432).

 -- Ben Howard <email address hidden> Thu, 05 Nov 2015 09:50:57 -0700

[Brian Murray (brian-murray)]

Hello Ben, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Pitt (pitti)]

I adjusted the autopkgtest-cloud infrastructure for this change yesterday, as this caused some fallout.

I now created a custom cloud image for precise with the -proposed update, and tests that ran against it confirm that there now are apt sources for restricted/multiverse/backports, etc:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-precise/precise/i386/b/bbswitch/20151117_222222@/log.gz

[Martin Pitt (pitti)]

This actually did introduce quite a major behaviour change: "apt-get source foo" will now download the -backports version, *not* the version that you would get with "apt-get install".

[Ben Howard (darkmuggle-deactivatedaccount)]

The behavior change was both expected and calculated. The problem here is that the cloud images and cloud-init driven boots would have different package sources than a bare-metal desktop or server install.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.24

---------------
cloud-init (0.6.3-0ubuntu1.24) precise; urgency=medium

  * d/patches/lp-1506244-azure-ssh-key-values.patch: AZURE: Add support
    and preference for fabric provided public SSH public key values over
    fingerprints (LP: #1506244).

 -- Ben Howard <email address hidden> Tue, 17 Nov 2015 10:02:08 -0700

[Scott Moser (smoser)]

just to collect other references to fallout, this change is at least partially responsible for bug 1523608 (https://bugs.launchpad.net/maas/+bug/1523608/comments/20)

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2370