[SRU] cloud-init archive template should match Ubuntu Server

Bug #1177432 reported by Ben Howard on 2013-05-07
30
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-init
Undecided
Unassigned
cloud-init (Ubuntu)
High
Unassigned
Precise
High
Unassigned
Trusty
High
Unassigned
Vivid
High
Unassigned
Wily
High
Unassigned
Xenial
High
Unassigned

Bug Description

[SRU Justification]
Ubuntu Cloud Images are inconsistent with desktop and bare-metal server installations since backports, restricted and multiverse are not enabled. This is effected via cloud-init that uses a template to select an in-cloud archive.

[FIX] Make the cloud-init template match that of Ubuntu-server.

[REGRESION] The potential for regression is low. However, all users will experience slower fetch times on apt-get updates especially on slower or high latency networks.

[TEST]
1. Build image from -proposed
2. Boot up image
3. Confirm that "sudo apt-get update" pulls in backports, restricted and multiverse.

Backports are currently not enabled in the cloud-init template. This is needed in order to get the backport kernels on cloud images.

Related bugs:
 * bug 997371: Create command to add "multiverse" and "-backports" to apt sources
 * bug 1513529: cloud image built-in /etc/apt/sources.list needs updating

Related branches

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1177432/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
Matt Heard (f-matt) on 2013-05-15
affects: ubuntu → cloud-init (Ubuntu)
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cloud-init (Ubuntu):
status: New → Confirmed
Scott Moser (smoser) wrote :

== summary ==
fix this problem by creating tools to manage /etc/apt/sources.list

== details ==
I realize that cloud-init is just kind of caught in the middle here because of the template that it has, and the fact that that template has not been updated to keep in sync with changes made elsewhere to the default sources list.

I personally prefer the smaller sources.list that is present in the cloud images. For reference, the differences between a server install of 13.04 cd and cloud-init template at the moment are:
 - cloud-init: (12 total entries, 2 x 3 x 2)
    {deb, deb-src} $release{"",-updates,-security} {main, universe}
 - install cd: (32 total entries, 2 x 4 x 4)
    {deb, deb-src} $release{"",-updates,-security,-backports} {main,restricted,universe,multiverse}

Ignoring multiarch, and translations, which are also multipliers, That means 20 more GETs every time 'apt-get update' is done. That is nothing to be scoffed at.

I put a gist at https://gist.github.com/smoser/5586288 which uses chdist to easily time runs of 'apt-get update'. The differences are very real.
In quick tests on raring, I was seeing a refresh taking 20 seconds for "full list" versus 8 seconds for cloud-images list against a locally warmed cache.

This is partially because apt is doing the GET serially, but even the cpu decompression does take time.

My basic issue boils down to this:
 * I expect that when people launch an instance, they immediately install software on it. In order to do that, they need to run 'apt-get update'.
 * If apt-get update goes from taking 8 seconds to 20 seconds, that is a significant increase in total time to functional system.

I'd prefer that we solve this *not* by making cloud-init have a large amount of additional sources. If I were to suggest making a change, it would be to remove deb-src entries from /etc/apt/sources.list to make it smaller.

My suggestion is that we fix this problem by adding tools that manipulate /etc/apt/sources.list or sources.list.d similar to 'apt-add-repository', but that support aliases like 'backports', 'partner', 'multiverse'.
Example usage:
 - aptsources --no-source set main universe
 - aptsources remove multiverse

Cloud-init can then expose such simple instructions in its cloud-config syntax.

Scott Moser (smoser) wrote :

2 comments:
 a.) bug 997371 is now fixed, so you should be able to do: apt-add-repository multiverse
 b.) some relevant discussion at https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2013-May/014503.html

description: updated
Scott Moser (smoser) on 2014-06-03
Changed in cloud-init (Ubuntu):
importance: Undecided → Low
Stéphane Graber (stgraber) wrote :

Just ran into this issue again today when testing the LXD backport.

This is pretty annoying as it's causing a completely different user experience for cloud users compared to regular distro users.

Also, at least in trusty, apt-add-repository doesn't know how to add backports, so a user will need to manually update /etc/apt/sources.list and then run apt-get update which makes installing backports significantly harder than it should be.

Dustin Kirkland  (kirkland) wrote :

This is bad. Please fix this asap.

Changed in cloud-init (Ubuntu):
importance: Low → High
status: Confirmed → Triaged
assignee: nobody → Ben Howard (utlemming)
Changed in cloud-init (Ubuntu):
status: Triaged → In Progress
summary: - Enable backports in cloud-init archive template
+ [SRU] Enable backports in cloud-init archive template
description: updated

Ben,

To confirm the behavior, will the default be kept for selecting back-ported packages? Specifically, in order to install a back-ported image a cloud-image user would have to be explicit in choosing to install from back-port, ie
  $ sudo apt-get install <package-name>/trusty-backports

ref = https://help.ubuntu.com/community/UbuntuBackports

Antonio Rosales (arosales) wrote :

--- Correction ---

Ben,

To confirm the behavior, will the default be kept for selecting back-ported packages? Specifically, in order to install a back-ported _package_ a cloud-image user would have to be explicit in choosing to install from back-port, ie
  $ sudo apt-get install <package-name>/trusty-backports

ref = https://help.ubuntu.com/community/UbuntuBackports

In re to #7/#8, the handling of selections from backports will _NOT_ be changed.

Users will have to specifically opt-in to selecting packages.

i.e to get regular ansible:
   $ sudo apt-get install ansible

Whereas are the following examples are valid for pulling from backports
  $ sudo apt-get -t trusty-backports install ansible
  $ sudo apt-get install lxd/trusty-backports

Antonio Rosales (arosales) wrote :

Re comment 9, thanks for confirming @utlemming.

-Antonio

Scott Moser (smoser) wrote :
description: updated
description: updated
Scott Moser (smoser) wrote :

I've committed the change to trunk that actually puts the template now inline with what is provided by 14.04 and 15.10 server iso installs (as seen in those attachments above).

I've also added tools to https://gist.github.com/smoser/5586288 and some more information. This illustrates well my desire for '--if-necessary' as requested in bug 1429285.

To summarize the changes below. That is roughly 450k additional download data (compressed size). Most notable is a stark doubling of GETs from 27 to 58:

    ## old cloud sources.list after "apt-get clean"
    $ ./summarize-http out.d/cloud/clean/update.err
    ...
    data bytes: 14431826
    total GETS: 27
    200: 16
    404: 11

    ## old cloud sources.list immediately repeated 'apt-get update'
    $ ./summarize-http out.d/cloud/noclean/update.err
    ...
    data bytes: 0
    total GETS: 27
    304: 16
    404: 11

    ## ISO sources.list after "apt-get clean"
    $ ./summarize-http out.d/cd/clean/update.err
    ...
    data bytes: 14893753
    total GETS: 58
    200: 37
    404: 21

    ## ISO sources.list immediately repeated 'apt-get update'
    $ ./summarize-http out.d/cd/noclean/update.err
    ...
    data bytes: 0
    total GETS: 58
    304: 37
    404: 21

Hello Ben, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
Chris J Arges (arges) wrote :

Hello Ben, or anyone else affected,

Accepted cloud-init into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1091-0ubuntu10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Vivid):
status: New → Fix Committed
Changed in cloud-init (Ubuntu Wily):
status: New → Fix Committed
Chris J Arges (arges) wrote :

Hello Ben, or anyone else affected,

Accepted cloud-init into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1149-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Marking as verification-failed pending a rebase on the new upstream trunk.

tags: added: verification-failed
removed: verification-needed
Scott Moser (smoser) on 2015-11-05
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.7~bzr1154-0ubuntu1

---------------
cloud-init (0.7.7~bzr1154-0ubuntu1) xenial; urgency=medium

  * New upstream snapshot.
    * create the same /etc/apt/sources.list that is present in default server
      ISO installs. This change adds restricted, multiverse, and -backports
      (LP: #1177432).

 -- Scott Moser <email address hidden> Thu, 05 Nov 2015 12:10:00 -0500

Changed in cloud-init (Ubuntu Xenial):
status: In Progress → Fix Released
summary: - [SRU] Enable backports in cloud-init archive template
+ [SRU] cloud-init archive template should match Ubuntu Server
description: updated
Changed in cloud-init (Ubuntu Precise):
importance: Undecided → High
Changed in cloud-init (Ubuntu Vivid):
importance: Undecided → High
Changed in cloud-init (Ubuntu Trusty):
importance: Undecided → High
Changed in cloud-init (Ubuntu Wily):
importance: Undecided → High

Hello Ben, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-failed
tags: added: verification-needed
Chris J Arges (arges) wrote :

Hello Ben, or anyone else affected,

Accepted cloud-init into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1091-0ubuntu11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Chris J Arges (arges) wrote :

Hello Ben, or anyone else affected,

Accepted cloud-init into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1149-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Verified changes enable multiverse, restricted and backports in cloud images.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.7~bzr1149-0ubuntu3

---------------
cloud-init (0.7.7~bzr1149-0ubuntu3) wily; urgency=medium

  * d/patches/lp-1177432-same-archives-as-ubuntu-server.patch: use the
    same archive pockets as Ubuntu Server (LP: #1177432).

 -- Ben Howard <email address hidden> Thu, 05 Nov 2015 12:41:19 -0700

Changed in cloud-init (Ubuntu Wily):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.7~bzr1091-0ubuntu11

---------------
cloud-init (0.7.7~bzr1091-0ubuntu11) vivid; urgency=medium

  * d/patches/lp-1177432-same-archives-as-ubuntu-server.patch: use the
    same archive pockets as Ubuntu Server (LP: #1177432).

 -- Ben Howard <email address hidden> Thu, 05 Nov 2015 12:36:21 -0700

Changed in cloud-init (Ubuntu Vivid):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.14

---------------
cloud-init (0.7.5-0ubuntu1.14) trusty; urgency=medium

  * d/patches/lp-1177432-same-archives-as-ubuntu-server.patch: use the
    same archive pockets as Ubuntu Server (LP: #1177432).

 -- Ben Howard <email address hidden> Thu, 05 Nov 2015 09:50:57 -0700

Changed in cloud-init (Ubuntu Trusty):
status: Fix Committed → Fix Released

Hello Ben, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Precise):
status: New → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Martin Pitt (pitti) wrote :

I adjusted the autopkgtest-cloud infrastructure for this change yesterday, as this caused some fallout.

I now created a custom cloud image for precise with the -proposed update, and tests that ran against it confirm that there now are apt sources for restricted/multiverse/backports, etc:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-precise/precise/i386/b/bbswitch/20151117_222222@/log.gz

tags: added: verification-done-precise
removed: verification-needed
Martin Pitt (pitti) wrote :

This actually did introduce quite a major behaviour change: "apt-get source foo" will now download the -backports version, *not* the version that you would get with "apt-get install".

The behavior change was both expected and calculated. The problem here is that the cloud images and cloud-init driven boots would have different package sources than a bare-metal desktop or server install.

tags: added: verification-done
removed: verification-done-precise
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.24

---------------
cloud-init (0.6.3-0ubuntu1.24) precise; urgency=medium

  * d/patches/lp-1506244-azure-ssh-key-values.patch: AZURE: Add support
    and preference for fabric provided public SSH public key values over
    fingerprints (LP: #1506244).

 -- Ben Howard <email address hidden> Tue, 17 Nov 2015 10:02:08 -0700

Changed in cloud-init (Ubuntu Precise):
status: Fix Committed → Fix Released
Scott Moser (smoser) wrote :

just to collect other references to fallout, this change is at least partially responsible for bug 1523608 (https://bugs.launchpad.net/maas/+bug/1523608/comments/20)

Joshua Powers (powersj) on 2017-09-11
Changed in cloud-init:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers