cloud images should be built with the same /etc/apt/sources.list as server images

Bug #1513529 reported by Scott Moser on 2015-11-05
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-images
Undecided
Dan Watkins
livecd-rootfs (Ubuntu)
High
Dan Watkins
Nominated for Trusty by Robert C Jennings
Xenial
Medium
Unassigned

Bug Description

[Impact]

In systems created from cloud images where cloud-init does not run (or has not yet run), /etc/apt/sources.list does not have the same contents as it has in a server install. This inconsistency can cause image modification/bootstrap to fail.

[Test Case]

Examine the cloud image built from livecd-rootfs, and confirm that the sources.list matches that in a server ISO install.

[Regression Potential]

Limited; the majority of cloud image usage does invoke cloud-init, which already writes out a source.list that matches the server ISO. This change aligns non-cloud-init usage to the existing norm.

[Original Report]

When we were poking around under bug 1177432 we found that /etc/apt/sources.list that is built into the image does not match that of an installed system. This /etc/apt/sources.list is used if cloud-init did not re-write the file (such as woudl be used if you mounted the image directly and ran apt-get update).

Example showing the problem:
$ qemu-img create -f qcow2 -b wily/release-20151029/ubuntu-15.10-server-cloudimg-amd64-disk1.img /tmp/disk.img
$ sudo mount-image-callback /tmp/disk.img --read-only chroot _MOUNTPOINT_ cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ wily main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ wily-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu/ wily-security main restricted universe multiverse

What we'd like to see here is exactly what we just added to cloud-init.
For reference, as attached to bug 1177432, see trusty [1] and wily [2] examples.

For reference, MAAS installed systems end up getting the built-in /etc/apt/sources.list with the ubuntu mirrors updated. So this change after making it all the way through will result in maas images having the same list as ISO installed systems also.

--
[1] https://launchpadlibrarian.net/224142290/trusty-sources.list
[2] https://launchpadlibrarian.net/224142308/wily-sources.list

Related bugs:
 * bug 1177432 [SRU] Enable backports in cloud-init archive template

Related branches

Scott Moser (smoser) on 2015-11-05
summary: - cloud image built in /etc/apt/sources.list needs updating
+ cloud image built-in /etc/apt/sources.list needs updating

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1513529/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
Scott Moser (smoser) on 2015-11-05
Changed in ubuntu:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Ben Howard (utlemming)
Scott Moser (smoser) on 2015-11-05
description: updated
affects: ubuntu → uec-live-build
Ben Howard (utlemming) wrote :

This was fixed released. Desktop/server/cdimages all use the same repositories.

Changed in uec-live-build:
status: Confirmed → Fix Released
Scott Moser (smoser) wrote :

I've just verified that this is not fixed in wily or xenial, so i've re-opened the bug and also added livecd-rootfs as that is being used for building xenial+.

Changed in uec-live-build:
assignee: Ben Howard (utlemming) → nobody
status: Fix Released → Confirmed
Changed in livecd-rootfs (Ubuntu):
status: New → Confirmed
importance: Undecided → High
Andres Rodriguez (andreserl) wrote :

It seems that this goes all the way back to trusty images as well, as the Juju team reported that backports are not enabled by default.

Changed in uec-live-build:
status: Confirmed → Invalid
Changed in cloud-images:
milestone: none → y-2016-05-05
Changed in cloud-images:
assignee: nobody → Dan Watkins (daniel-thewatkins)
no longer affects: uec-live-build
summary: - cloud image built-in /etc/apt/sources.list needs updating
+ cloud images should be built with the same /etc/apt/sources.list as
+ server images
Changed in cloud-images:
status: New → In Progress
Changed in livecd-rootfs (Ubuntu):
assignee: nobody → Dan Watkins (daniel-thewatkins)
status: Confirmed → In Progress
Changed in cloud-images:
milestone: y-2016-05-05 → y-2016-05-19
Michael Foord (mfoord) wrote :

Note that this is blocking testing for juju and maas.

Dan Watkins (daniel-thewatkins) wrote :

I've confirmed that trusty-backports is enabled for trusty images; I'm downloading a precise image to verify now.

The xenial change will (I think) need to be SRU'd in to livecd-rootfs, I'll look at that next.

Dan Watkins (daniel-thewatkins) wrote :

Confirmed that this is fixed in precise.

Changed in cloud-images:
milestone: y-2016-05-19 → y-2016-06-02
tags: added: awaiting-canonical
Changed in cloud-images:
milestone: y-2016-06-02 → y-2016-06-16
Changed in livecd-rootfs (Ubuntu):
status: In Progress → Fix Committed
status: Fix Committed → Fix Released
Dan Watkins (daniel-thewatkins) wrote :

So to fix the specific issue that was blocking juju and MAAS, -backports has been enabled in precise and trusty images.

yakkety images will have the same sources.list as in the server installation (albeit via a one-time sync).

As xenial has been released with a difference between the sources.lists and there isn't a currently compelling reason to modify it under users, we won't make the change there. If, in the future, there is a specific reason to revisit this decision, we will be happy to do so then.

Changed in cloud-images:
status: In Progress → Fix Released
Scott Moser (smoser) wrote :

I consider the fact that xenial is the *only* differently behaving release to be compelling.

$ for r in precise trusty xenial yakkety zesty; do n=bp-$r; echo ==== $r =====; lxc init ubuntu-daily:$r $n >/dev/null 2>&1 && lxc file pull $n/etc/apt/sources.list $r.list; pastebinit $r.list ; grep backports $r.list; done
==== precise =====
http://paste.ubuntu.com/23513229/
deb http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse
==== trusty =====
http://paste.ubuntu.com/23513230/
deb http://archive.ubuntu.com/ubuntu trusty-backports main restricted universe multiverse
==== xenial =====
http://paste.ubuntu.com/23513231/
==== yakkety =====
http://paste.ubuntu.com/23513232/
## Also, please note that software in backports WILL NOT receive any review
deb http://archive.ubuntu.com/ubuntu/ yakkety-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ yakkety-backports main restricted universe multiverse
==== zesty =====
http://paste.ubuntu.com/23513233/
## Also, please note that software in backports WILL NOT receive any review
deb http://archive.ubuntu.com/ubuntu/ zesty-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ zesty-backports main restricted universe multiverse

Changed in livecd-rootfs (Ubuntu Xenial):
status: New → Confirmed
importance: Undecided → Medium
Stéphane Graber (stgraber) wrote :

This is causing problem for LXD users on Xenial as our instructions say to use "apt install -t xenial-backports lxd" to get the latest non-LTS release of LXD on 16.04 LTS.

We've had report from MAAS users that xenial-backports isn't enabled on otherwise clean systems. This appears to be because MAAS instructs cloud-init not to regenerate sources.list, therefore using the original sources.list from the cloud image.

We'd very much appreciate some consistency there as having to instruct folks to manually mangle their sources file isn't exactly ideal, especially when it's unneeded in 90% of the case.

description: updated

Hello Scott, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in livecd-rootfs (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed
Scott Moser (smoser) wrote :

This does not look fixed to me.

$ /tmp/check-backports.sh /tmp/apt-sources.d
==== precise =====
build_name: server
serial: 20170327
http://paste.ubuntu.com/24267929/
deb http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse
==== trusty =====
build_name: server
serial: 20170323
http://paste.ubuntu.com/24267930/
deb http://archive.ubuntu.com/ubuntu trusty-backports main restricted universe multiverse
==== xenial =====
build_name: server
serial: 20170327
http://paste.ubuntu.com/24267932/
==== yakkety =====
build_name: server
serial: 20170327
http://paste.ubuntu.com/24267933/
## Also, please note that software in backports WILL NOT receive any review
deb http://archive.ubuntu.com/ubuntu/ yakkety-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ yakkety-backports main restricted universe multiverse
==== zesty =====
build_name: server
serial: 20170327
http://paste.ubuntu.com/24267934/
## Also, please note that software in backports WILL NOT receive any review
deb http://archive.ubuntu.com/ubuntu/ zesty-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ zesty-backports main restricted universe multiverse
FAILED. missing backports: xenial

tags: added: verification-failed
removed: verification-needed

I've realised that asking Scott to validate this wasn't the right call because he doesn't have a way of easily building images from -proposed. I have now built an image using the -proposed livecd-rootfs, and I see the following line in /etc/apt/sources.list of the produced image:

deb http://archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse

So this is indeed working.

tags: added: verification-done
removed: verification-failed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.408.9

---------------
livecd-rootfs (2.408.9) xenial; urgency=medium

  [ Daniel Watkins ]
  * Don't overwrite the default sources.list in cloud images.
  * Replace sources.list generated using COMPONENTS with the sources.list from
    an Ubuntu Server installation (i.e. with all components enabled, and all
    deb-src lines commented). LP: #1513529.

  [ Chris Glass ]
  * Fix the manifest generation in OVA files so that ovf files don't have
    double extensions. (LP: #1627931)
  * Fix the OVF's metadata to include Ubuntu specific identifiers and
    descriptions instead of the generic Linux ones. (LP: #1656293)

  [ Daniel Watkins ]
  * Add replace_grub_root_with_label function thereby consolidating multiple
    uses of the same calls to sed.

  [ Robert C Jennings ]
  * ubuntu-cpc: Remove redundant copy of grub files. (LP: #1637290)

 -- Robert C Jennings <email address hidden> Thu, 23 Mar 2017 14:40:59 -0400

Changed in livecd-rootfs (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers